During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of magnitude $10^{-5}$ for the DQN with PER, whereas the mean squared loss was an order of magnitude $10^{-2}$.

Do the smaller training errors have any effect on executing the final policy learned by the DQN?


I think it says something about the training progress, while another approach you can make sure is to look at the gradient norm. Sometimes, the training loss is really noisy while the gradient norm is much more clear.

  • $\begingroup$ u mean the gradient of the weights of the model ? what would the gradient norm say about the training ? $\endgroup$ – calveeen May 12 '20 at 0:51
  • $\begingroup$ well smaller norm would indicate that training is about to converge i guess. At a local minima, all the dimensions of a gradient would be 0. $\endgroup$ – SpiderRico Jun 11 '20 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.