I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an input to RNN for encoding the state of the system, I cannot change the weights of my network in the middle of the episode. Is that correct? Otherwise, the hidden state vectors will be different when the weights are changed.
Does that mean that the use of RNN in RL has to store the entire episode before the weights can be changed?
How does then one take into account the hidden states in RNN for RL? Are there any good tutorials on RNN-RL?