I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared value error objective function in (9.1):

$\bar{VE}(\boldsymbol{w}) = \sum_{s \in S} \mu(s)[v_{\pi}(s) - \hat{v}(s,\boldsymbol{w})]^2$. (9.1)

$\boldsymbol{w}$ is a vector of the parameterized function $\hat{v}(s,\boldsymbol{w})$ that approximates the value function $v_{\pi}(s)$. $\mu(s)$ is the fraction of time spent in $s$, which measures the "importance" of state $s$ in $\bar{VE}(\boldsymbol{w})$.

In (9.4), it states an update rule of $\boldsymbol{w}$ by gradient descent: $\boldsymbol{w}_{t+1} = \boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$. (9.4)

I have two questions regarding (9.4).

  1. Why $\mu(s)$ is not in (9.4)?
  2. Why is it the "minus" instead of "+" in (9.4)? In other words, why is it $\boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$ instead of $\boldsymbol{w} +\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$?
  • $\begingroup$ I think your equation number 9.4 is not correct? There is indeed + sign in Sutton's book and also $\nabla{V(s,w)}$ at the end. $\endgroup$ – Swakshar Deb Oct 7 '20 at 20:42
  • $\begingroup$ They are giving same importance to all the states, so there is no $\mu(s)$ $\endgroup$ – Swakshar Deb Oct 7 '20 at 20:45
  • $\begingroup$ @SwaksharDeb the + sign is in (9.5) because it expands the gradient equation. But why it is a - sign in the first place in (9.4)? $\endgroup$ – JYY Oct 7 '20 at 20:50
  • $\begingroup$ @SwaksharDeb So in (9.4) it "assumes" all the states are equally important, otherwise $\mu(s)$ will appear in the equation? $\endgroup$ – JYY Oct 7 '20 at 20:51
  • $\begingroup$ This is a stochastic gradient descent update. I think your problem is related to gradient descent. $\endgroup$ – Swakshar Deb Oct 7 '20 at 20:52
  1. $\mu(s)$ is not in equation (9.4) because we are assuming that the examples by which we update our parameter $w$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $w$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by $\alpha$.

  2. The minus is there because we are performing gradient descent. For more information on this, see e.g. the wikipedia page


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.