I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last turn, the game ends, and the non-zero reward is given to the agent. There are no more observations to be made and there is no next state $s'$ to store in the experience tuple. How should the final states be handled?

  • $\begingroup$ This question seems to be a duplicate of this. $\endgroup$ – nbro Nov 1 '20 at 19:14

You do not store a terminal state as $s$ in the replay table because by definition its value is always $0$, and there is no action, reward or next state. There is literally nothing to learn.

However, you may find it useful to store information that $s'$ is actually a terminal state, in case this is not obvious. That is typically achieved by storing an additional done boolean component. This is useful, because it allows you to branch when calculating the TD target g:

s, a, r, next_s, done = replay_memory_sample()
if done:
  g = r
  g = r + gamma * max( q(s') )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.