Tag Info

This is a case of overfitting the Q function leading to compounding errors when selecting actions. You have been training your neural network as function approximator for too long on the same data distribution, so the neural network loses it's ability to generalize and slowly starts overfitting, i.e. learns the data exactly as it is or at least very closely....

The overestimation comes from the random initialisation of your Q-value estimates. Obviously these will not be perfect (if they were then we wouldn't need to learn the true Q-values!). In many value based reinforcement learning methods such as SARSA or Q-learning the algorithms involve a $\max$ operator in the construction of the target policy. The most ...