1
$\begingroup$

I'm aware that we back-propagate after computing the loss between:

The Neural Network Q values and the Target Network Q values

However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.

Suppose one epoch is run and the reward is +10, surely we need to update the parameters using this too to tell the Network to push the probability of these actions, given these parameters up.

How does the algorithm know +10 is good? Suppose the reward range is -10 for loss and +10 for win.

$\endgroup$
1
$\begingroup$

However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.

Yes. That is all it needs to do because we have defined the policy around the Q values like so:

$$\pi(s) = \text{argmax}_a \hat{q}(s,a,\theta)$$

Where $\theta$ is the neural network weights.

Therefore, if the estimates of Q are approximately the same as the action value of the optimal policy, the policy in DQN is approximately the optimal policy.

How does the algorithm know +10 is good?

It does not, at least not directly. The algorithm knows, approximately, what the action values are if it acts consistently with their current estimates by always choosing the maximising action at each step.

The learning process will learn that +10 is relatively good in your scenario because it never finds anything better when exploring.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.