I asked a similar question a few days back here, but since no one replied, I thought I should subdivide my question further.
My understanding of RNNs is as follows,
Suppose I have a standard MLP. To get to a simple RNN as outlined by the original paper by Elman, I need only introduce a new weight matrix $R$ to one of the layers. Suppose the recurrent layer was layer $i$, then
In the graphical picture of neural networks, this means that at the recurrent layer, every neuron is connected to every other neuron in the same layer. However, on implementing this, I find that there is no performance improvement over the standard MLP. Further, this interpretation means that every timestep is fed forward, and back propagated, through the entire network (including the non-recurrent layers). This seems to disagree with the Keras implementation, where the entire sequence is fed through only the recurrent layer and one vector output is given to the next layer.
I'm pretty sure I'm misunderstanding something, but I'm not sure what. It would be nice if someone could point it out.