Problem Statement:

I am given 2 sets of images. All the images in both sets are without annotations and labels.

First set : a set of images of the grocery store shelves (captured in the grocery stores).

Second set: a set of close-up images of the products kept on those store shelves.

What I am trying to achieve:

Goal: For every product image, I want to find the location of that product in all shelf images in which it appears.

For example,

Input Product image Product image

Output Corresponding Shelf image Shelf image

My approach:

  1. For each product image, first find all the shelf image(s) which contain that product.
  2. Then predict a bounding box by finding the location of the product in the shelf image.

I am using YOLOv5 for this task but I am not sure how should I start off given that I have to do it without annotations or labels.

I have come across terms like Zero-shot learning, self-supervised Object Detection, etc. but I haven't been able to figure out their use case as a starting point.

There's a similar question asked on StackOverflow but I am not sure the answer to it solves the problem.

  • $\begingroup$ Can you please put your specific question in the title? "Object Detection without annotations and labels" is not a question and it's also not very specific. $\endgroup$
    – nbro
    Commented Apr 12, 2022 at 14:20
  • $\begingroup$ @nbro Done! If I may ask, does that make it more clear now? For a detailed description of the problem, please read the explanation down there that I have provided along with the newly edited title. $\endgroup$ Commented Apr 12, 2022 at 14:43

2 Answers 2


You can formulate your problem as a template matching (TM) one. This task is similar to object detection, but you don't need to label anything. There are 2 images: the template (in your case, the template is the image of a product) and the image where you want to search the template (the image with the shelf of products). There are different TM algorithms. For example, OpenCV offers a simple one with different similarity measures.

From my limited experience with TM, it works well only if the objects in both the template and the other image are in a plane parallel to image plane. This is more or less your case (at least, in the images you're showing us), so I expect that TM will work reasonably well with those 2 specific images, but you should first resize the first image so that there are no padding around the object (the chocolate bar) - in a way, this is like labeling. If this feasible, then you should try TM.

  • $\begingroup$ I am given a Hint: Use Visual Similarity. Maybe I am wrong but I believe TM isn't of much use here because time spent resizing all the product images is as good as labeling, so please usher me to a path like where I have a pre-trained model on a similar dataset and now I would like to freeze some layers and fine-tune the model on a new dataset which comes with a problem that it does not have labels or annotations. So how should I proceed from there? Note : You can assume I have no experience with self-supervising the pre-trained model, or maybe something better that you can suggest? $\endgroup$ Commented Apr 12, 2022 at 19:20
  • $\begingroup$ I don't know who gave you that hint, but template matching uses some kind of "visual similarity". There might be a tool that attempts to automatically resize the image to fit the object in it. So, resizing might be less laborious than labelling and maybe you don't have to do a lot manual work. Moreover, you probably don't have to be super precise when resizing. You can still leave some padding/spacing around the object and it might still work. Having said that, it might be possible to use a pre-trained model to do your task, but I don't know which one - I am not an expert in this topic. $\endgroup$
    – nbro
    Commented Apr 12, 2022 at 20:29

For something like this you may just want to compare the image features directly, rather than trying to fanangle your data into a class based segmentation model. There are a number of very fast feature detection algorithms, like SIFT, SURF, ORB, and BRIEF, that don't necessitate the use of a neural network at all.

OpenCV - Feature Matching

The features locked away in the hidden layer of a CNN are gold for this sort of thing as well, and chopping the head off of a pretrained neural network to get at those conv layers, and then using those in a manner similar to SIFT features, may improve accuracy while accomplishing the same end.

Paper - Deep Semantic Feature Matching

Either way, you are going to be doing some kind of similarity search here, rather than a classification task. You want to use methods that see if the important bits of image A exist in image B, and any attempt to do that via classification models would create so many classes as to be unusable. Any method used to accomplish this task is going to have to rely on similarity of image features in one way or another.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .