I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared value error objective function in (9.1):

$\bar{VE}(\boldsymbol{w}) = \sum_{s \in S} \mu(s)[v_{\pi}(s) - \hat{v}(s,\boldsymbol{w})]^2$. (9.1)

$\boldsymbol{w}$ is a vector of the parameterized function $\hat{v}(s,\boldsymbol{w})$ that approximates the value function $v_{\pi}(s)$. $\mu(s)$ is the fraction of time spent in $s$, which measures the "importance" of state $s$ in $\bar{VE}(\boldsymbol{w})$.

In (9.4), it states an update rule of $\boldsymbol{w}$ by gradient descent: $\boldsymbol{w}_{t+1} = \boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$. (9.4)

I have two questions regarding (9.4).

  1. Why $\mu(s)$ is not in (9.4)?
  2. Why is it the "minus" instead of "+" in (9.4)? In other words, why is it $\boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$ instead of $\boldsymbol{w} +\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$?
  • $\begingroup$ I think your equation number 9.4 is not correct? There is indeed + sign in Sutton's book and also $\nabla{V(s,w)}$ at the end. $\endgroup$ Oct 7, 2020 at 20:42
  • $\begingroup$ They are giving same importance to all the states, so there is no $\mu(s)$ $\endgroup$ Oct 7, 2020 at 20:45
  • $\begingroup$ @SwaksharDeb the + sign is in (9.5) because it expands the gradient equation. But why it is a - sign in the first place in (9.4)? $\endgroup$
    – JYY
    Oct 7, 2020 at 20:50
  • $\begingroup$ @SwaksharDeb So in (9.4) it "assumes" all the states are equally important, otherwise $\mu(s)$ will appear in the equation? $\endgroup$
    – JYY
    Oct 7, 2020 at 20:51
  • $\begingroup$ This is a stochastic gradient descent update. I think your problem is related to gradient descent. $\endgroup$ Oct 7, 2020 at 20:52

1 Answer 1

  1. $\mu(s)$ is not in equation (9.4) because we are assuming that the examples by which we update our parameter $w$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $w$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by $\alpha$.

  2. The minus is there because we are performing gradient descent. For more information on this, see e.g. the wikipedia page


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .