How does the regression layer in the localization network of a spatial transformer work?

I am trying to understand the spatial transformer network mentioned in this paper https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf. I am clear about the last two stages of the spatial transformer i.e. the grid generator and sampler. However I am unable to understand the localization network which outputs the parameters of the transformation that is applied to the input image. So here are my doubts.

1. Is the network trained on various affine/projective transforms of the input or only the standard input with a standard pose?
2. If the answer to question 1 is no, then how does the regression layer correctly regress the values of the transformation applied to the image? In other words how does the regression layer know what transformation parameters are required when it has never seen those inputs before?

Thanks in advance.

1 Answer

1. Localization network is not trained separately on special transform of input. It's just a part of feed forward network which is trained as a whole, with normal backpropagation.

2. It's just part of the network which affect the loss function. As in any backpropagation loss function propagate back gradient, which backpropagate through final part of the network, backpropagate through differentiable sampler(that is non-trivial part, which use transfromation produced by localization subnetwork) and after that backpropagate into localization part

3. Whole approach of spatial transformers could be in doubt now. There were some anecdotal evidences that it was not working on less trivial tasks (private communications, was not working for me either)

• Thanks for the reply. However my doubt still stands. How does the regression layer output the exact transform parameters in a previously unseen image? It seems magic to me. Am I missing something about regression which the localization network uses? Feb 26, 2020 at 7:03
• It's called "regression layer" because it produce not discrete set of probabilities but continuous parameters. It's just common fully connected layer. Nothing special about it. And it seems whole method work only on transformation of simple flat images like road signs. It's not so good for affine approximation, not a magical solution Feb 26, 2020 at 7:07