I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing.

Is this a sign that it is diverging?

Or should be just be looking at the mean Q value, which has a slight uptrend?

enter image description here

1 There are 2 different colored lines because the initial training (in orange) was stopped at around 1.3M time steps, and resumed (in blue) from a checkpoint at around 1.1M time steps

2 Plots are from Tensorboard, visualizing data generated by Ray/RLlib

3 Epsilon starts at 1.0 and anneals to 0.02 over 10000 time steps. The sudden increase in magnitude of Q-values appear to come after resuming from checkpoint, but might just be a coincidence.

After training for more steps...

enter image description here



You must log in to answer this question.