Object Classification: How to decide which detected region is a RoI for classification?

Question

I am working on a project where I am working on the Flickr-47 dataset to do logo detection and classification. My approach is to first finetune a YOLO v5 model with high recall to detect as many "logos" as it possibly can and then to classify the detected regions using some sort of feature extraction.

Here is an example of the actual image and what the model detects as logos.

Ground Truth Annotations:

YOLO detections:

My question is because there are instances as can be seen above where the number of regions detected by the model can be less or more than the actual regions to be detected in the ground truth image, how do I decide which ones to keep for classification?

P.S. I know I can apply non-max suppression to remove overlaps but the above can still persist if the YOLO detector finds something worth a logo, but isn't. I can also modify the IoU threshold and confidence threshold during detection inference, but I want to be able to extract as much as the YOLO model detects, and keep the filtering at a later stage.

Any ideas?

Answer 1

I am not too familiar with YOLO but from having a brief look at it and the images you posted, they seem to be detecting letters (which is quite ironic because on StackOverflow this is what people usually have trouble with e.g. https://stackoverflow.com/questions/43003369/can-we-use-yolo-to-detect-and-recognize-text-in-a-image )

But anyway one thing I know the way in the past, from my own personal experience with object recognition, is usually an AI / Robot best detects things in terms of shapes, usually a Triangle,.Circle and square then cross-reference that within a dataset of similarities, e.g. four cylinders and a square = stool. You could do a similar thing here perhaps, if object / image contains circle alone, or square alone or triangle alone or combined with the latter 3 shapes then such and such % chance a logo. The issue would come when you have logos that are all letters. I can see its done similar in the images scoring less on the fonts and more on the shapes which makes some sense to what I am saying. If you have any more context to those images I could perhaps answer better in that respect. So basically I would score shapes higher like in the images,that are shapes, and text lower at first but then...

I am not sure if you can train YOLO but that would seem a logical way forward in terms of differing between shapes/logos and fonts, but then you would have to score higher on say bigger sized fonts would be a good way to detect text based logos. So I would train it until it has a good basis for sizing of fonts and shape differences overall.

I am probably oversimplifying it here, never used YOLO but I always start simple before scaling up and it never failed me in the past and before learning something new. Apologies if this is the not the answer you are looking for, I can't comment yet.