How are the observations stored in the RNN that encodes the state?

Question

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an input to RNN for encoding the state of the system, I cannot change the weights of my network in the middle of the episode. Is that correct? Otherwise, the hidden state vectors will be different when the weights are changed.

Does that mean that the use of RNN in RL has to store the entire episode before the weights can be changed?

How does then one take into account the hidden states in RNN for RL? Are there any good tutorials on RNN-RL?

Answer 1

This research question seems to be analyzed in further details here (section 3) - https://openreview.net/pdf?id=r1lyTjAqYX.

Usually, a sequence is taken as a state to be fed into RNN to compute the final hidden state. One can then ask what initial state should the RNN be seeded with? This paper analyzes three methods with respect to the seed -

• zero initialization: When the RNN is initialized with the zero state
• burn-in: when the sequence is prepended by some preceding observations for RNN to learn a good initial state
• storing the initial hidden state: When the hidden state at the beginning of the sequence is stored