I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end, and so far my agent seems to be learning without terminal flags. I was wondering how important this feature is, as it's in all the pseudocode but doesn't seem to be necessary in my implementation.


It's an important feature, and you drop it at the risk of the agent failing to learn successfully.

The difference between the TD target without the terminal flag

$$G_t = R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$$

and with the terminal flag applied to $S_{t+1} = S_T$

$$G_t = R_{t+1}$$

is important whenever $Q(S_{T}, a')$ might be evaluated as non-zero. The true action value of the terminal state is always defined as zero.

In some cicumstances, values based on a function approximator estimate could be far from zero - if at any point the estimator moves significantly away from zero then this could lead to value estimates diverging. This can be a problem at any time, because the estimator is usually not trained on actions taken from the terminal state - due to them never being observed. In principle you could add training data to keep the function approximator in line with this estimate, but as it is an absolute value it is far more common (when using approximators such as neural networks) to change the TD target as above.

Double Q learning improves estimates of bootstrap values by removing some of the maximisation bias from the next action value. This may help in your case, if the expected future rewards from the terminal state remain close to zero.

I would expect Double Q learning to work OK without using terminal state informaton, and without adding "mitigating" training to keep terminal state estimates close to zero, if rewards are not sparse, and do not increase close to the terminal states. That would mean the estimator should recognise expected future reward becomes closer to zero as the agent approaches a terminal state, thus would not take much extrapolation to predict close to zero for the terminal state - close enough that it does not create runaway feedback or influence the optimal policy much.

With sparse rewards, or with significantly high rewards close to the terminal state (e.g. a goal state which the agent must reach to gain all the reward), then using the terminal flag in the normal way becomes more important. It is unlikely that double Q learning would help much in that case. However, it may still be possible to find the optimal policy but with highly inaccurate action values (e.g. action values could all be double what they should be).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.