I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last turn, the game ends, and the non-zero reward is given to the agent. There are no more observations to be made and there is no next state $s'$ to store in the experience tuple. How should the final states be handled?

  • $\begingroup$ This question seems to be a duplicate of this. $\endgroup$
    – nbro
    Nov 1, 2020 at 19:14

1 Answer 1


You do not store a terminal state as $s$ in the replay table because by definition its value is always $0$, and there is no action, reward or next state. There is literally nothing to learn.

However, you may find it useful to store information that $s'$ is actually a terminal state, in case this is not obvious. That is typically achieved by storing an additional done boolean component. This is useful, because it allows you to branch when calculating the TD target g:

s, a, r, next_s, done = replay_memory_sample()
if done:
  g = r
  g = r + gamma * max( q(s') )

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .