2
$\begingroup$

From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for storing the action $a$.

As I recall, the reward $r$ and the next state $s'$ are both used to calculate the target values. We can then compare these target values to the output we get when we do a forward-pass using state $s$ It seems to me that the stored action $a$ is not required for this process to work; or am I missing something? Why would we store the action $a$ if it isn't used in the training process itself?

Please, forgive me if this question has been answered before. I looked, but was unable to find anything other than generic explanations as to what experience replay is and why we do it.

$\endgroup$
3
$\begingroup$

We need to store the action $a$ as it tells us the action that we took in the state that we are backing up.

Suppose we are in state $s$ and we take action $a$, then we will receive a reward $r$ and next state $s'$. The goal of RL, and in particular DQN (I mention DQN as it is the first algorithm that comes to mind when I think of a replay buffer but it is of course not the only algorithm to make use of one), is that we are trying to learn optimal state-action value functions $Q(s, a)$. We thus want our value function to be able to predict $y = r + \gamma \max_{a'}Q(s', a')$, i.e. given $s$ and $a$ we want to be able to predict $y$. As you can see, we need to know which action we took in state $s$ so that we can train our value function to approximate $y$, and the value function clearly depends on $a$, hence we need to also store $a$ in our experience tuple.

$\endgroup$
3
  • $\begingroup$ Perhaps I now understand where my mistake was. Do I understand correctly, that for calculating the loss and performing gradient descent, we do not actually compare all of the Q-values, but only the one that corresponds to $Q(s,a)$ to its target value? Until now, I was under the impression we would do something like mean-squared-error, where we compare all the Q-values to their "labels" or rather the targets we calculated using $s'$ and $r$. $\endgroup$ Feb 23 at 12:14
  • $\begingroup$ You're correct that we only update the Q-value for one action. I posted an answered here that seems like it might explain this in more detail. $\endgroup$ Feb 23 at 14:36
  • 1
    $\begingroup$ I see. I think that answers my question, then :-) Thank you very much $\endgroup$ Feb 23 at 14:51
2
$\begingroup$

The goals of experience replay as first proposed by Lin (1992) and more recently applied successfully in the DQN algorithm by Mnih et al. (2013) are to break temporal correlations of updates and to prevent forgetting of experiences that might be useful later on.

To meet these goals, the replay buffer should store tuples required in the learning step.

Most works that use experience replay, including those mentioned before, learn (to approximate) the Q function, i.e. $Q(s,a)$. Clearly, this function depends on the sampled action $a$.

If $a$ is not used at all in the training process then it would not need to be stored in the replay buffer. In such a scenario, however, the problems that motivate the replay buffer may not be present in the first place.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.