I am trying to understand the spatial transformer network mentioned in this paper https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf. I am clear about the last two stages of the spatial transformer i.e. the grid generator and sampler. However I am unable to understand the localization network which outputs the parameters of the transformation that is applied to the input image. So here are my doubts.
- Is the network trained on various affine/projective transforms of the input or only the standard input with a standard pose?
- If the answer to question 1 is no, then how does the regression layer correctly regress the values of the transformation applied to the image? In other words how does the regression layer know what transformation parameters are required when it has never seen those inputs before?
Thanks in advance.