What is the purpose of storing the action $a$ within an experience tuple?

Question

From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for storing the action $a$.

As I recall, the reward $r$ and the next state $s'$ are both used to calculate the target values. We can then compare these target values to the output we get when we do a forward-pass using state $s$ It seems to me that the stored action $a$ is not required for this process to work; or am I missing something? Why would we store the action $a$ if it isn't used in the training process itself?

Please, forgive me if this question has been answered before. I looked, but was unable to find anything other than generic explanations as to what experience replay is and why we do it.

Answer 1

We need to store the action $a$ as it tells us the action that we took in the state that we are backing up.

Suppose we are in state $s$ and we take action $a$, then we will receive a reward $r$ and next state $s'$. The goal of RL, and in particular DQN (I mention DQN as it is the first algorithm that comes to mind when I think of a replay buffer but it is of course not the only algorithm to make use of one), is that we are trying to learn optimal state-action value functions $Q(s, a)$. We thus want our value function to be able to predict $y = r + \gamma \max_{a'}Q(s', a')$, i.e. given $s$ and $a$ we want to be able to predict $y$. As you can see, we need to know which action we took in state $s$ so that we can train our value function to approximate $y$, and the value function clearly depends on $a$, hence we need to also store $a$ in our experience tuple.

Answer 2

The goals of experience replay as first proposed by Lin (1992) and more recently applied successfully in the DQN algorithm by Mnih et al. (2013) are to break temporal correlations of updates and to prevent forgetting of experiences that might be useful later on.

To meet these goals, the replay buffer should store tuples required in the learning step.

Most works that use experience replay, including those mentioned before, learn (to approximate) the Q function, i.e. $Q(s,a)$. Clearly, this function depends on the sampled action $a$.

If $a$ is not used at all in the training process then it would not need to be stored in the replay buffer. In such a scenario, however, the problems that motivate the replay buffer may not be present in the first place.