How does YOLO detect the object when the object is in multiple grid cells?

Question

I have been reading various articles and watching videos on YouTube, but I can't seem to understand one thing.

How does YOLO make a bounding box for an object if it is in multiple grid cells? For example, in the picture given below, how does it predicts the bounding box for the classes, because they fall in multiple cells? How does it know what object is in a grid cell even when it sees a small part of it?

It's been very difficult for me to get these answers.

Answer 1

Lets say you split the image by 13x13 (SxS) cells. For each grid cell, you predict 5 (older YOLO) or 9 (newer versions) bounding boxes.

These (anchors) are supposed to have various shapes and scales, so they vertical, horizontal and squared rectangle at small scale, and middle scale and bigger scale.

So for each cell (13x13) and for each anchor (9) , you predict (width offset ,height offset,center x,center y, objectness, class1 ,class2,... class k ). You actually predict scaling of the width and height of the box, so you can end up with variable sizes.

Ojectness marks if there is any object in this cell for this anchor, if not, it is discarded later.

Since you end up with lot of predicted bounding boxes of various shapes, you need to filter them out, this is called non maximum supression. For each anchor you take class with highest confidence.

EDIT:

So to answer your questions, If an object is in multiple cells, it is contained in multiple predicted boxes, and you take the one with maximum confidence and discard others.

Each cell does not see only part of the image, it sees the whole image. Before you get to 13x13 cells, you have 50 or more layers of conv & max pooling layers, which gives each cell receptive field of the whole image.

Answer 2

TL;TR

The location of the input that a neuron sees is called receptive field.

grid cell is merely used to demonstrate the concept of anchor. It has nothing to do with the receptive field of the network structure. The actual receptive field size (RF) depends on the paramters of yolo structure, different size of the yolo network actually has different RF. The RF is definitely not the full size of the input image, but it is also way larger than the grid cell. Usually several times bigger. The smaller the feature map, the bigger the ratio of RF/cell

For yolov8 tiny, the RF of P3 is 12 times of P3 grid cell size. P4 is 7x, P5 is 4x.

yolo use each P3-P5 to predict exactly one bonding box per neuron. For P3 it is 80x80 neurons, for P4 it is 40x40 neurons, for P5 it is 20x20 neurons.

The size of the object can be predict for each neuron is not constrained at all. It only limit the location of the object. see detail section for more discussion.

During training, one grid cell only provide a single box, but same ground truth box can be assigned to multiple grid cells. Thus during inference it is possible to have multiple grid cells per feature map predict a single instance.

To handle this multiple-to-one problem, we need a postprocess step to select one box for each instance. During inference, the model predict additional score: class-quality score which roughly means the class probability multiple by box quality. This is then used as likely-hood score or confidence for corresponding box:

1. we first use a threshold of the confidence to filter out the low quality bbox.
2. for each bbox it choose the argmax of class confidence as the instance class.
3. If we are lucky there is no further postprocess is required. However, it is very common, that we still have some redundant boxes per instance. There are boxes per each class have large overlaps. To reduce the redundancy we use a heuristic algorithm called NMS. It is heuristic, because in high dense scenario it is possible to have multiple instance closely overlapped, but with NMS we can only detect one of these instances.
4. the NMS basically applied to each class separately. It basically start from high confidence box then remove the boxes has lower confidence but high overlap ratio

some details about the relationship of grid cell, predict box and receptive field.

A neuron can only predict a box contains the grid cell centre, but the box size can be much larger or much smaller. The extreme case is demonstrated in following image. The predicted box can be as big as the image itself. There is no limit of the predicted box size relative to either grid cell or receptive field.

So why we allow it to predict a box larger than the input (receptive field)? Because the network can "guess" the actual box via seeing partial of the object. For example, even though we can only see part of the bicycle, we can still estimate how big the actual bicycle is:

more details about the receptive field

Let's take the modern yolov8 as example. To simplify stuff we only look at the backbone. It is a CNN structure.

In CNN the receptive field only changed when the conv kernel size > 1 or stride size > 1:

• The conv has k=1, s=1, is only a channel mixer which does not blend any lateral spatial info.
• when k>1 each neuron see kxk input
• when s>1, the current layer skip some input while scanning. Thus equivalently it makes the next layer's neuron seeing wider input.

Thus the RF can be calculate as: $$ RF_{l} = RF_{l-1} + (K_{l} - 1) \times \prod_{j=1}^{l-1} S_{j} $$

Where:

• $RF_{l}$ is the receptive field of layer $l$.
• $K_{l}$ is the kernel size of layer $l$.
• $S_{j}$ is the stride of layer $j$.

Here is the actual structure overview. Focus on the left side backend part.

the C2f is tricky, it contains n bottlnecks each of the bottleneck has 4 conv modules. For tiny model the d = 0.33

We can roughly see it as

• stem conv
• C1 conv
• C2 C2f n=1
• conv
• C3 C2f n=2
• conv
• C4 C2f n=2
• conv
• C2f C2f n=1
• C5 (SPPF) (9)

So the kernel size and stride for each layer can be represent as a list of list: blocks

# each bottleneck blcok has 4 conv that k>1
btn = [(3,1)]*4
# stem, C1 (1), C2 (2), conv, C3 (4), conv, C4 (6), conv, C2f, C5 (SPPF) (9)
idx_Cs = [1,2,4,6,9]
blocks = [
  [(3,2)], # conv
  [(3,2)], # conv
  btn*1, # C2f n=3*d
  [(3,2)], # conv
  btn*2, # C2f n=6*d
  [(3,2)],
  btn*2, # C2f n=6*d
  [(3,2)],
  btn*1, # C2f n=3*d
  [(3,1), (5,1), (5,1), (5,1), (3, 1)] # SPPF
]

For the backbone we already have recptive field for each feature map like following:

C1 has RF= 7x7
C2 has RF= 15x15
C3 has RF= 35x35
C4 has RF= 55x55
C5 has RF= 83x83

However, we still have FPN which continues to fuse the feature map to form P3, P4, P5. This further increases the receptive field, because for each Pn, the longest path is longer than C5. I will not put details here but I can show the final results:

P3 has RF= 99x99
P4 has RF= 111x111
P5 has RF= 123x123

detection heads then add another 2 convs that can change RF. Resulting the final RF:

DET_P3 has RF= 103x103
DET_P4 has RF= 115x115
DET_P5 has RF= 127x127

Remember, for P3, P4, and P5, the reducing factor is 8, 16, 32. This means the grid cell size in pixel is: 8x8, 16x16 and 32x32. We can see they are far smaller than the actual receptive fields.

Now you also understand the limit of the yolo structure, it usually struggling to predict big object which cover most of the image, since the RF of P5 does not cover the entire input. This is also why there is enhanced version that has P6 layer to add a bigger feature map.

You can find the code used to compute the RF here.