# REINFORCE with Baseline update rule

I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton:

I do not quite understand the update rule for $$w$$:

$$w = w + \alpha \delta \nabla \hat{v}(S_t, w)$$

If I understand correctly, this would be the update rule, if $$\delta$$ would be the MSE:

$$\delta = (G-\hat{v}(S_t, w))^2$$

$$\nabla\delta = -2(G-\hat{v}(S_t, w))\nabla \hat{v}(S_t, w)$$

which would give the same update as shown in the algorithm, when we include the $$2$$ into the learning rate $$\alpha$$

$$w = w + \alpha \delta \nabla \hat{v}(S_t, w)$$

However, in the algorithm they do not use the MSE for $$\delta$$, but simply

$$\delta = G-\hat{v}(S_t, w)$$

$$\nabla\delta = - \nabla \hat{v}(S_t, w)$$

which would yield a different update rule (compared to the one in the algorithm)

$$w = w + \alpha \nabla \hat{v}(S_t, w)$$

What am I missing?

The value $$\delta$$ is already representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $$\hat{v}$$ to get a gradient for the weights is applying the chain rule.
Your equations would work and be compatible with the book, if instead of trying to redefine $$\delta$$, you created the implied loss function $$L = \frac{1}{2}(\hat{v}(S_t, w) - G)^2$$ and set $$\delta = \frac{d L}{d \hat{v}}$$