# What information should be cached in experience replay for actor-critic?

Experience replay is buffer (or a "memory") of transactions $$e_t = (s_t, a_t, r_t, s_{t+1})$$.

actor loss (parameterized by $$\theta$$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by $$w$$) $$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}).$$

As I see it, there are two more elements that need to be saved for later use:

1. The expected Q value at the time $$t$$: $$Q_w(s_{t},a_{t})$$

2. The log probability for action $$a_t$$: $$log[\pi_\theta(s_t,a_t)]$$

If they are not saved, how will we be able to later calculate the loss for learning? I didn't see anywhere stating to save those, and I must be missing something.

Do these elements need to be saved or not?

The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. 100 instances) from the replay memory, and by having the states you can input them to the respective network and have the $$Q(s)$$ for every state in your batch. Then you estimate the loss and run backpropagation and the networks' weights are getting updated. You continue gathering experience by interacting with the environment and after a threshold that you specify you repeat the cycle by re-sampling a new batch from the memory.