How do I calculate the error during the training phase for deep reinforcement learning models?

Deep reinforcement learning is not supervised learning as far as I know. So how can the model know whether it predicts right or wrong? In literature, I find that the "actual" Q-value is calculated, but that sounds like the whole idea behind deep RL is obsolete. How could I even calculate/know the real Q-value if there is not already a world model existing?


Yes, reinforcement learning is very different from supervised learning, the policy (what you call a model) does not know if its predicting right or wrong, or if its taking the correct action or not. In RL there is no concept of "the right action", everything is evaluated through the reward function.

Also there are no ways to compute the ground truth Q-values, if you had that then there is no need to do RL.

In RL you should not think like in supervised learning, there are no error metrics, everything is evaluated on how much accumulated reward the agent receives over an episode.

  • 1
    $\begingroup$ Actually, RL and SL aren't so different or unrelated. In fact, every SL problem can be cast as an RL one. However, your suggestions are usually valid. Anyway, I think you should address more directly the question "How could I even calculate/know the real Q-value if there is not already a world model existing?" by showing that you don't need to know the true value e.g. in Q-learning. $\endgroup$ – nbro Feb 19 '20 at 18:01
  • $\begingroup$ @nbro I did not claim that RL and SL were unrelated, but they are very different. Try doing the opposite, casting a RL problem as a supervised learning one, and you will see the difference. In RL you do not have access to "the right answer". $\endgroup$ – Dr. Snoopy Feb 19 '20 at 20:34
  • $\begingroup$ Yes, I agree with you. I just wanted to point out that they aren't completely unrelated, as one may interpret when reading your first sentence. $\endgroup$ – nbro Feb 19 '20 at 20:49
  • $\begingroup$ So, how are the weights updated then? Given a state, when the neural net leads just to a random action, how will the best policy be learned over time? $\endgroup$ – MScott Feb 22 '20 at 11:05
  • $\begingroup$ @MScott That is a question on its own, there are many techniques to do it (Q-Learning, Policy gradients, etc), a neural network is just used as a function approximation inside those frameworks. $\endgroup$ – Dr. Snoopy Feb 22 '20 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.