How does non-max suppression work when one or multiple bounding boxes are predicted for the same object?

Question

My understanding of how non-max suppression works is that it suppresses all overlapping boxes that have a Jaccard overlap smaller than a threshold (e.g. 0.5). The boxes to be considered are on a confident score (may be 0.2 or something). So, if there are boxes that have a score over 0.2 (e.g. the score is 0.3 and overlap is 0.4) the boxes won't be suppressed.

In this way, one object will be predicted by many boxes, one high score box, and many low confident score boxes, but I found that the model predicts only one box for one object. Can someone enlighten me?

I currently viewing the ssd from https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection

Here is the code.

#Finding Jaccap Overlap and sorting scotes
class_scores, sort_ind = class_scores.sort(dim=0, descending=True)
class_decoded_locs = class_decoded_locs[sort_ind]  # (n_min_score, 4)
overlap = find_jaccard_overlap(class_decoded_locs, class_decoded_locs)  
suppress = torch.zeros((n_above_min_score), dtype=torch.uint8).to(device)  

for box in range(class_decoded_locs.size(0)):
# If this box is already marked for suppression
    if suppress[box] == 1:
    continue
    suppress = torch.max(suppress, overlap[box] > max_overlap)
    suppress[box] = 0

Answer 1

I might be able to help with the theory, but the coding... it is a non standard API such as Tensorflow or Pytorch (it might be custom code for what I can tell).

The key element here is that the bouding boxes are removed only if they hold a prediciton for the same class that the box that is overlapping with (but with less confidence, that is why it gets removed).

Here is an example, where we have:

• Two classes $c \in [c_1, c_2] = [$ "star"$, $ "moon" $]$
• Three bounding boxes

The blue bounding boxes holds prediction for the class $c_1$ so their predictions are $p(c_1)_{box1} = 0.8$ and $ p(c_1)_{box2} = 0.9$. On the other hand, the green box holds a prediction for the class $c_2$.

The three boxes are highly overlaping so the overlap between any box $x$ with any box $y$ will be above the IoU threshold: $IoU(box_x, box_y) > 0.5$. So in principle all boxes are suceptible to be removed.

However the NMS only applies for boxes predicting the same class (in the case the blue one). So the NMS algorihtm is: if the boxes are overlapping, $IoU(box_1, box_2) > 0.5$, which is true, remove all non maximal class probability boxes. Said differently, take just the box with highest $p(c_1)$ and remove the rest. So the $box_1$ with class probability $p(c_1) = 0.8$ would be removed.

So what happens with green box? Isn't it overlapping as well? Yes, but consider that the green box is not trying to predict the same object, is trying to predict another object, $c_2$, which happens to be very close to the first object, $c_1$. This way object detectors support detection of different overlapping objects.