Experience replay is buffer (or a "memory") of transactions $e_t = (s_t, a_t, r_t, s_{t+1})$.

The equations for calculating the loss in actor critic are an

actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by $w$) $$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}).$$

As I see it, there are two more elements that need to be saved for later use:

  1. The expected Q value at the time $t$: $Q_w(s_{t},a_{t})$

  2. The log probability for action $a_t$: $log[\pi_\theta(s_t,a_t)]$

If they are not saved, how will we be able to later calculate the loss for learning? I didn't see anywhere stating to save those, and I must be missing something.

Do these elements need to be saved or not?


1 Answer 1


The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. 100 instances) from the replay memory, and by having the states you can input them to the respective network and have the $Q(s)$ for every state in your batch. Then you estimate the loss and run backpropagation and the networks' weights are getting updated. You continue gathering experience by interacting with the environment and after a threshold that you specify you repeat the cycle by re-sampling a new batch from the memory.

Just a suggestion: Start moving towards Asynchronous/synchronous methods for RL and use one network with different "heads" for Actor and Critic. Then you use one loss function plus the experience that now is collected from multiple instances of your agent-environment interaction (in contrast to your description in which one agent collects and stores experience from one instance).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .