What is a heatmap in the CornerNet paper?

Question

I have been working on understanding how CornerNet works, but I couldn't figure out a few parts about the architecture.

First, the authors mention that there are 3 distinct parts to be predicted as a heatmap, embedding, and offset.

Also, in the paper, it is stated that the network was trained on the COCO dataset, which has bounding box and class annotations.

As far as I am concerned, since CornerNet is based on detecting the top-left and bottom-right corners, the ground-truth labels for heatmap should be composed of top-left and bottom-right pixel locations of bounding boxes with the class score (but I might be wrong). What is the heatmap used for?

Moreover, for the embedding part, authors used the pull&push loss at the ground-truth pixel locations to find out which corner pairs belong to which object, but I don't understand how to backpropagate this loss. How do I back-propagate the embedding loss?

Answer 1

Heatmap in the sense of Corner net is the heatmap of the pooled corner values. As discussed in the paper, there is a corner pooling operation that gives you the vector values for a pixel point being a corner(it may or not). The output from the corner pooling is CxHxW. Then to generate a heat map you train the network similar to the Grad-CAM method. Training for heatmap generation means that we are going to tell how much a given pixel point has a weight to be a corner point. i.e if it has a high weight it will generate more colored values on that pixel area.

The heatmap is used for the prediction of corner points. That means if you know which pixel has a higher weight then that pixel will have a higher probability to hold the top-right or bottom-left corner of the given bounding box.