From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for storing the action $a$.
As I recall, the reward $r$ and the next state $s'$ are both used to calculate the target values. We can then compare these target values to the output we get when we do a forward-pass using state $s$ It seems to me that the stored action $a$ is not required for this process to work; or am I missing something? Why would we store the action $a$ if it isn't used in the training process itself?
Please, forgive me if this question has been answered before. I looked, but was unable to find anything other than generic explanations as to what experience replay is and why we do it.