I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode?
- Mean Q-values are calculated on a fixed set of states by taking mean of max Q-values for each state.
- By convergence I mean that the plot of mean Q-values converge to some level (those values does not increase to infinity).
It can be seen in here (page 7) that mean Q-values converge and average rewards plot is quite noisy. I get similar results and in tests, the best policy is where the peaks are during training (average reward plot). I don't understand why don't I get better average scores over time (and better policies) when Q-values converge.