1
$\begingroup$

How do I calculate the error during the training phase for deep reinforcement learning models?

Deep reinforcement learning is not supervised learning as far as I know. So how can the model know whether it predicts right or wrong? In literature, I find that the "actual" Q-value is calculated, but that sounds like the whole idea behind deep RL is obsolete. How could I even calculate/know the real Q-value if there is not already a world model existing?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Yes, reinforcement learning is very different from supervised learning, the policy (what you call a model) does not know if its predicting right or wrong, or if its taking the correct action or not. In RL there is no concept of "the right action", everything is evaluated through the reward function.

Also there are no ways to compute the ground truth Q-values, if you had that then there is no need to do RL.

In RL you should not think like in supervised learning, there are no error metrics, everything is evaluated on how much accumulated reward the agent receives over an episode.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Actually, RL and SL aren't so different or unrelated. In fact, every SL problem can be cast as an RL one. However, your suggestions are usually valid. Anyway, I think you should address more directly the question "How could I even calculate/know the real Q-value if there is not already a world model existing?" by showing that you don't need to know the true value e.g. in Q-learning. $\endgroup$
    – nbro
    Feb 19, 2020 at 18:01
  • $\begingroup$ @nbro I did not claim that RL and SL were unrelated, but they are very different. Try doing the opposite, casting a RL problem as a supervised learning one, and you will see the difference. In RL you do not have access to "the right answer". $\endgroup$
    – Dr. Snoopy
    Feb 19, 2020 at 20:34
  • $\begingroup$ Yes, I agree with you. I just wanted to point out that they aren't completely unrelated, as one may interpret when reading your first sentence. $\endgroup$
    – nbro
    Feb 19, 2020 at 20:49
  • $\begingroup$ So, how are the weights updated then? Given a state, when the neural net leads just to a random action, how will the best policy be learned over time? $\endgroup$
    – MScott
    Feb 22, 2020 at 11:05
  • $\begingroup$ @MScott That is a question on its own, there are many techniques to do it (Q-Learning, Policy gradients, etc), a neural network is just used as a function approximation inside those frameworks. $\endgroup$
    – Dr. Snoopy
    Feb 22, 2020 at 12:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .