I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing.
Is this a sign that it is diverging?
Or should be just be looking at the mean Q value, which has a slight uptrend?
1 There are 2 different colored lines because the initial training (in orange) was stopped at around 1.3M time steps, and resumed (in blue) from a checkpoint at around 1.1M time steps
2 Plots are from Tensorboard, visualizing data generated by Ray/RLlib
3 Epsilon starts at 1.0 and anneals to 0.02 over 10000 time steps. The sudden increase in magnitude of Q-values appear to come after resuming from checkpoint, but might just be a coincidence.
After training for more steps...