I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? Or there is something wrong in my implementation?
My exploration vs explotation greedy strategy goes from 1.0 to 0.1 in 1 million steps (as DeepMind does), my learning rate is 0.00025 and my gamma 0.99. I read here that
"The mean Q-values should smoothly converge towards a value proportionnal to the mean expected reward."
So, it's my agent expecting a negative reward? If so, how can i fix it?
Here is a graph of the first training session:
You can see how the Q-values tend to converge near-zero after about 1300 episodes (1120000 steps aproximately). Actually it's showing values like -0.0117, -0.0145, etc.
Also, the agent seems very "static" after epsilon gets near 0.1, and when it reaches it doesn't move so much. (I'm training with