For an image (of any object), how to find its location in the other image(s) which contains it, given there are no labels or annotations for any image

Question

Problem Statement:

I am given 2 sets of images. All the images in both sets are without annotations and labels.

First set : a set of images of the grocery store shelves (captured in the grocery stores).

Second set: a set of close-up images of the products kept on those store shelves.

What I am trying to achieve:

Goal: For every product image, I want to find the location of that product in all shelf images in which it appears.

For example,

Input Product image

Output Corresponding Shelf image

My approach:

1. For each product image, first find all the shelf image(s) which contain that product.
2. Then predict a bounding box by finding the location of the product in the shelf image.

I am using YOLOv5 for this task but I am not sure how should I start off given that I have to do it without annotations or labels.

I have come across terms like Zero-shot learning, self-supervised Object Detection, etc. but I haven't been able to figure out their use case as a starting point.

There's a similar question asked on StackOverflow but I am not sure the answer to it solves the problem.

Answer 1

You can formulate your problem as a template matching (TM) one. This task is similar to object detection, but you don't need to label anything. There are 2 images: the template (in your case, the template is the image of a product) and the image where you want to search the template (the image with the shelf of products). There are different TM algorithms. For example, OpenCV offers a simple one with different similarity measures.

From my limited experience with TM, it works well only if the objects in both the template and the other image are in a plane parallel to image plane. This is more or less your case (at least, in the images you're showing us), so I expect that TM will work reasonably well with those 2 specific images, but you should first resize the first image so that there are no padding around the object (the chocolate bar) - in a way, this is like labeling. If this feasible, then you should try TM.

Answer 2

For something like this you may just want to compare the image features directly, rather than trying to fanangle your data into a class based segmentation model. There are a number of very fast feature detection algorithms, like SIFT, SURF, ORB, and BRIEF, that don't necessitate the use of a neural network at all.

OpenCV - Feature Matching

The features locked away in the hidden layer of a CNN are gold for this sort of thing as well, and chopping the head off of a pretrained neural network to get at those conv layers, and then using those in a manner similar to SIFT features, may improve accuracy while accomplishing the same end.

Paper - Deep Semantic Feature Matching

Either way, you are going to be doing some kind of similarity search here, rather than a classification task. You want to use methods that see if the important bits of image A exist in image B, and any attempt to do that via classification models would create so many classes as to be unusable. Any method used to accomplish this task is going to have to rely on similarity of image features in one way or another.