In Faster R-CNN, how can I get the predicted bounding box given the neural network's output?

Question

The RPN loss in Faster RCNN paper is

$$ L({p_i}, {t_i}) = \frac{1}{N_{cls}} \sum_{i} L_{cls}(p_i,p_i^*) + \lambda \frac{1}{N_{reg}} \sum_i p_i^* L_{reg}(t_i, t_i^*) $$

For regression problems, we have the following parametrization

$$t_x=\frac{x - x_a}{w_a}, \\ t_y=\frac{y−y_a}{h_a}, \\ t_w= \log \left( \frac{w}{w_a} \right),\\ t_h= \log \left(\frac{h}{h_a} \right)$$

and the ground-truth labels are

$$t_x^*=\frac{x^* - x_a}{w_a},\\ t_y^*=\frac{y^*−y_a}{h_a}, \\ t_w^*= \log \left( \frac{w^*}{w_a} \right), \\ t_h^*= \log \left(\frac{h^*}{h_a} \right)$$

where

• $x$ and $y$ are the two coordinates of the center, $w$ the width, and $h$ the height of the predicted box.

• $x$ and $y$ are the two coordinates of the center, $w$ the width, and $h$ the height of the anchor box.

• $L_{reg}(t_i, t_i^*) = R(t_i − t_i^*)$, where $R$ is a robust loss function (smooth $L_1$)

These equations are unclear to me, so here are my two questions.

1. How can I get the predicted bounding box given the neural network's output?

2. What exactly is the smooth $L_1$ here? How is it defined?

Answer 1

If I understood well you have 2 questions.

• How to get the bounding box given the network output
• What Smooth L1 loss is

The answer to your first question lies in the equation (2) in the section 3.2.1 from the Faster R-CNN paper. As all anchor based object detector (Faster RCNN, YOLOv3, EfficientNets, FPN...) the regression output from the network are not the bounding box coordinates. The regression output predicts the shift of the predicted bounding box with respect to the selected anchor (all of these networks, use more than 1 anchor per location, check section 3.1.1 from the paper).

So basically what your network predict is $t_x, t_y, t_w, t_h$:

And the bounding box coordinates are $x, y, w, h$, and the anchor coordinates are $x_a, y_a, w_a, h_a$. So in order to compute $x, y, w, h$ from $t_x, t_y, t_w, t_h$, you just have to invert the equations above. However I think you could gain more intuition about it if you take your time and read the whole section 3.1 from the paper. I know sometimes is a pain, but you will grasp the high level concept.

With regard to your second question. Yes the loss is computed with the output from the network and the "coded" ground truth, meaning you compute the loss with the paramters $t$ (predicted) against $t^*$ (coded ground truth) instead for computing loss with the real coordinates of the bounding boxes (decoded output from the network). For the equation on Smooth L1 loss check this wonderful documentation.