I want to tackle the problem of detecting similar objects in an image. To illustrate the problem consider this photo of some Lego bricks as my "input":

enter image description here

The detection routine should identify similar objects. So for the given input, it should e.g. identify the following output:

enter image description here

So an object might appear none to multiple times in the input image. For example, there are only two bricks marked with a blue cross, but three bricks marked with a red cross.

It can be assumed that all objects are of similar kind, so e.g. only Lego bricks or a heap of sweets.

My initial idea is to apply a two-phased approach:

  1. Extract all objects in input image.

  2. Compare those extracted objects and find similar ones.

Is this a valid approach or is there already some standard way of solving this kind of problem? Can you give me some pointers how to solve this problem?


Disclaimer: I have never used Siamese networks

I would approach this problem in two steps:

First: train object detector and train it for eligible classes of object, for example using Yolo architecture.

You could use pretrained object detector and finetune it for your classes of objects.

Second: extract a lot of bounding boxes of eligible objects of the same classes from your dataset and train Siamese network on their subimages.

Siamese network output similarity measure between two objects.

Your pipleline would look like:

  1. Run object detetctor

  2. For each two pair of objects of the same class rescale bounding boxes and run Siamese network.

  3. Check if the similarity distance between two objects is less then threshold (tunable hyperparameter)

| improve this answer | |

Detection of similar objects is a categorization problem, but not of the simplistic kind that can be done in machine learning through training.

A probable effort reducing option is to exploit the idea that, once a similar pair can be found and a certainty can be attached to their similarity, the grouping algorithm is pure statistically derived algebra and arithmetic. A likely difficulty in completing this project with current technology is that translating an essentially conceptual idea into software is not a purely machine learning problem and not a purely logical one. It will require prowess across a number of AI disciplines.

Similarity in this project's context is not like numerical similarity, textual similarity, shape similarity, color similarity, or even visual similarity. The quantity of image pairs that would be needed to train a network deep enough to work from camera input directly into physical similarity is beyond the current convergence technology.

The visual systems of mammals probably don't make that direct transition in one functional step either. It is likely that 2D motion is translated into physical (3D) motion and that the ability for humans to be able to recognize object similarity in photos is an extension of that motion oriented translation of 2D into 3D. The features of edges, texture, shading, and the obscurity of far away objects by closer ones is an advantage in object recognition, not yet exploited by many AI software products.

Once there is a recognition of three dimensional form, optical characteristics, and texture, independent of proximity, orientation, partial obscurity, and lighting, then similarity detection can proceed in a way consistent with the project objective.

Another challenge in this project is that similarity is subjective. A person unfamiliar with traditional taxonomy would consider a coyote similar to a jaguar. Those with taxonomy education might consider a coyote similar to a dog and a jaguar similar to a cat. A bacteriologist might consider all four mammals similar.

If you solve both the mathematical imprecision of similarity and the 2D to 3D problems together and finish the project with the procurement of a system with reasonable accuracy and reliability, working out the expression of such a system mathematically and publishing would be indicated. Such would be worthy of notice.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.