I've read through a few papers on next frame prediction from a sequence of frames and several of them use spatial transformations (STNs). See this as an example. I want to know what are the pros and cons of using an STN to predict the next frame. Are there any assumptions that must be made about the data besides "Consecutive frames are all approximately affine transformations of each other"?
Using an STN alone to predict the next frame assumes that there is some linear translation between the current frame and the next frame. In some domains this is true but usually there are more complicated transitions from frame to frame (Eg. An occlusion, a new entity, light difference, etc). So, although STNs may be useful to resizing and translating inputs for a CNN, they should be used together with other techniques when predicting a new frame of a sequence.