# DQN Q-mean values converge negatively

I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? Or there is something wrong in my implementation?

My exploration vs explotation greedy strategy goes from 1.0 to 0.1 in 1 million steps (as DeepMind does), my learning rate is 0.00025 and my gamma 0.99. I read here that "The mean Q-values should smoothly converge towards a value proportionnal to the mean expected reward." So, it's my agent expecting a negative reward? If so, how can i fix it? Here is a graph of the first training session: You can see how the Q-values tend to converge near-zero after about 1300 episodes (1120000 steps aproximately). Actually it's showing values like -0.0117, -0.0145, etc. Also, the agent seems very "static" after epsilon gets near 0.1, and when it reaches it doesn't move so much. (I'm training with PongDeterministic-v4)

• Could you explain the reward scheme in PongDeterministic-v4? This is really critical to deciding whether DQN is being accurate, and environments can vary wildly in what the true value function is. What you typically want is to approximate the true value function for optimal play. There is no reason to expect that to have a specific value (such as near zero) - the "true" values are first and foremost a consequence of the environment and the policy. – Neil Slater Mar 27 '19 at 13:48
• @NeilSlater yes; first, the difference between standar enviroments in OpenAI-gym and Deterministic enviroments are that deterministic gets a fixed frameskip of 4 (as used in DeepMind Paper). the -v4 stands for the probability of repeat the last action (in v4 is 0; so it always follow the predicted action, as DeepMind did). Source here: github.com/openai/gym/issues/1280 The reward scheme is just making the ball pass behind his bar to score a point. The first who score 21 times wins. I implemented reward clipping but in Pong it doesn't affect (making the scores between -1 and 1). – JCP Mar 27 '19 at 14:46