In YOLO, when is $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what are the ground-truth labels for $x_i$ and $y_i$?

Question

I'm trying to implement a custom version of the YOLO neural network. Originally, it was described in the paper You Only Look Once: Unified, Real-Time Object Detection (2016). I have some problems understanding the loss function they used.

Basic information:

• An input image is divided into $S \times S$ grid (that gives the total of $S^2$ cells) and each cell predicts $B$ bounding boxes and $c$ conditional class probabilities. Each bounding box predicts $5$ values: $x,y,w,h,C$ (center of bounding box, width and height and confidence score). This makes the output of YOLO an $S \times S \times (5B*c)$ tensor.

• The $(x,y)$ coordinates are calculated relative to the bounds of the cell and $(w,h)$ is relative to the whole image.

• I understand that the first term penalizes the wrong prediction of the center of a bounding box; the 2nd term penalizes wrong width and height prediction; the 3rd term the wrong confidence prediction; the 4th is responsible for pushing confidence to zero when there is no object in a cell; the last term penalizes wrong class prediction.

My problem:

I don't understand when $\mathbb{1}^\text{obj}_{ij}$ should be $1$ or $0$. In the paper, they write (section 2.2. Training):

$\mathbb{1}_{i j}^{\mathrm{obj}}$ denotes that the $j$th bounding box predictor in cell $i$ is "responsible" for that prediction.

and they also write

Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box

• So, is it right that, for every object in the image, there should be exactly one pair of $ij$ such that $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$?

  • If this is correct, this means that the center of the ground truth bounding box should fall into $i$th cell, right?

  • If this is not the case, what are other possibilities when $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what ground truth labels $x_i$ and $y_i$ should be in these cases?

• Also, I assume that ground truth $p_i(c)$ should be $1$ if there is an object of class $c$ in the cell $i$, but what ground truth $p_i(c)$ should be equal to in case there are several objects of different classes in the cell?

Answer 1

The correct interpretation (based on the comment by the author of the question below):

Yep, you are right. It is actually only a single cell per object that contributes to the loss with $\mathbb{1}^\text{obj}_{ij}$ factor. That cell is identified as the one that contains the centre(oid) of the ground truth box of the corresponding object.

My original (incorrect) interpretation of the paper:

From what I read in the paper it is not a single bounding box for all cells in the grid for a particular object (your original suggestion).

Rather for every object and every cell that is assigned to this object only one bounding box contributes to the loss. The network generates $B$ bounding boxes for every cell in the grid, but we pick only one and only when that cell actually belongs to an object.

Every cell $i$, which is not a background cell, will have exactly one box $j$, such that $\mathbb{1}^\text{obj}_{ij} = 1$

This is based on the following paragraph from the paper:

Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box (i.e. has the highest IOU of any predictor in that grid cell).