2
$\begingroup$

Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggests an update rule (and the math behind) that is beyond my understanding?

enter image description here

Also, why do we need $I$?

$\endgroup$
  • $\begingroup$ In the pseudocode you provide, what you call the "squared distance" is also present. So, what's exactly confusing you, apart from the need for $I$ in the pseudocode? $\endgroup$ – nbro Jan 23 at 15:32
  • $\begingroup$ Please, fix your last comment to include the MathJax equations between the dollar symbols $. $\endgroup$ – nbro Jan 23 at 20:39
  • 1
    $\begingroup$ deleted the previous comment because could not edit it. instead of $w + \alpha \nabla (R + \hat{v}(S',w) - \hat{v}(S,w))^2$ why does the update of w look like $w + \alpha (R + \hat{v}(S',w) - \hat{v}(S,w)) \nabla \hat{v}(S,w)$? I think i don't get the math behind derivation – d56 12 mins ago Delete $\endgroup$ – d56 Jan 23 at 20:42
  • $\begingroup$ @nbro I think i get it. It is just an application of the derivative chain rule. First, we bring down the exponent, and then take derivative of the second term (we treat the first term as the constant). The questions remaing are: why do we treat the first term ($R + \hat{v}(S',w))$ as a constant although it is being estimated by the same neural network. Where does the exponent disappear to (is it being reduced by the learning rate $\alpha^\theta$)? What is the term $I$? $\endgroup$ – d56 Jan 28 at 18:45
  • $\begingroup$ Your ideas seem reasonable, but where did you see that the loss must be the square of that thing? If the gradient is taken with respect to the parameters $w$, then $R$ is a constant, but $\hat{v}(S', w)$ should not be a constant (as you say). The learning rate could indeed absorb the $2$. The $I$ is used in the update step of the policy parameters and it changes by the factor $\gamma$. I would need to review that part of the book to help you more. Maybe later. $\endgroup$ – nbro Jan 28 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.