Why do the object detection networks produce multiple anchor boxes per location?

Question

In various neural network detection pipelines, the detection works as follows:

1. One processes the input image through the pretrained backbone
2. Some additional convolutional layers
3. The detection head, where each pixel on the given feauture map predicts the following:
   • Offset from the center of the cell ($\sigma(t_x), \sigma(t_y)$ on the image)
   • Height and width of the bounding boxes $b_w, b_h$
   • Objectness scores (probability of object presence)
   • Class probabilities

Usually, detection heads produce not a single box, but multiple.

• The first version of YOLO - outputs 2 boxes per location on the feature map of size $7 \times 7$
• Faster R-CNN outputs 9 boxes per location
• YOLO v3 - outputs 9 boxes per pixel from the predefined anchors : (10×13),(16×30),(33×23),(30×61),(62×45),(59× 119), (116 × 90), (156 × 198), (373 × 326)

These anchors give the priors for the bounding boxes, but with the help of $\sigma(t_x), \sigma(t_y), b_w, b_h$ one can get any possible bounding box on the image for some pixel on the feature map.

Therefore, the network will produce plenty of redundant boxes, and a certain procedure - NMS suppresion has to be run over the bounding box predictions to select only the best.

Or the purpose of these anchors is to start from a prior, reshape and shift slightly the bounding box, and then compare with the ground truth.

Is it the case, that if one used only a single bounding box for detection - it would be hard to train the network to rescale the initial bounding to, say, 10 times, and produce some specific aspect ratio?

Answer 1

Yes, theoretically it is possible to learn the offsets to get any possible bounding box from only one anchor box. However, it is hard to learn such dramatic shifts and changes. Learning only small offsets from the prior is easier and tends to converge better.

In specific applications however, one might already know the typical size and ratio of objects, and that this is very similar for all of them. In such cases, one box per anchor can be enough to learn well.

Note that many redundant boxes are predicted anyway, even if only one anchor box is used per location, because there are usually many anchor locations distributed in a grid based fashion over the image. Therefore, NMS is a necessary step anyway and does not depend on having multiple boxes per anchor.