I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5
environment.
I want to check for overestimation, therefore I'm plotting the action-state values for each episode. After 400+ episodes, I noticed that sometimes my predicted action-state values are negative. There are no negative rewards in this environment, only a +1
reward for each step taken.
Is this normal or is it somehow a bug?
Here you can see what happens on different runs: the run indicated by the brown line shows negative values for a few steps.