# Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?

I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared value error objective function in (9.1):

$$\bar{VE}(\boldsymbol{w}) = \sum_{s \in S} \mu(s)[v_{\pi}(s) - \hat{v}(s,\boldsymbol{w})]^2$$. (9.1)

$$\boldsymbol{w}$$ is a vector of the parameterized function $$\hat{v}(s,\boldsymbol{w})$$ that approximates the value function $$v_{\pi}(s)$$. $$\mu(s)$$ is the fraction of time spent in $$s$$, which measures the "importance" of state $$s$$ in $$\bar{VE}(\boldsymbol{w})$$.

In (9.4), it states an update rule of $$\boldsymbol{w}$$ by gradient descent: $$\boldsymbol{w}_{t+1} = \boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$$. (9.4)

I have two questions regarding (9.4).

1. Why $$\mu(s)$$ is not in (9.4)?
2. Why is it the "minus" instead of "+" in (9.4)? In other words, why is it $$\boldsymbol{w} -\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$$ instead of $$\boldsymbol{w} +\frac{1}{2}\alpha \nabla[v_{\pi}(S_t) - \hat{v}(S_t,\boldsymbol{w})]^2$$?
• I think your equation number 9.4 is not correct? There is indeed + sign in Sutton's book and also $\nabla{V(s,w)}$ at the end. Oct 7 '20 at 20:42
• They are giving same importance to all the states, so there is no $\mu(s)$ Oct 7 '20 at 20:45
• @SwaksharDeb the + sign is in (9.5) because it expands the gradient equation. But why it is a - sign in the first place in (9.4)?
– JYY
Oct 7 '20 at 20:50
• @SwaksharDeb So in (9.4) it "assumes" all the states are equally important, otherwise $\mu(s)$ will appear in the equation?
– JYY
Oct 7 '20 at 20:51
• This is a stochastic gradient descent update. I think your problem is related to gradient descent. Oct 7 '20 at 20:52

1. $$\mu(s)$$ is not in equation (9.4) because we are assuming that the examples by which we update our parameter $$w$$, i.e. the frequency of which we will observe the states during online training, is the same. That is, it is a constant with respect to $$w$$ and since we are differentiating it can be somewhat disregarded as a constant of proportionality -- it essentially can be 'absorbed' by $$\alpha$$.