I'm training a simple deep q-learning algorithm with no experience buffer to solve the
I want to check for overestimation, therefore I'm plotting the action-state values for each episode. After 400+ episodes, I noticed that sometimes my predicted action-state values are negative. There are no negative rewards in this environment, only a
+1 reward for each step taken.
Is this normal or is it somehow a bug?
Here you can see what happens on different runs: the run indicated by the brown line shows negative values for a few steps.