I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton:

enter image description here

I do not quite understand the update rule for $w$:

$w = w + \alpha \delta \nabla \hat{v}(S_t, w) $

If I understand correctly, this would be the update rule, if $ \delta$ would be the MSE:

$\delta = (G-\hat{v}(S_t, w))^2$

$\nabla\delta = -2(G-\hat{v}(S_t, w))\nabla \hat{v}(S_t, w)$

which would give the same update as shown in the algorithm, when we include the $2$ into the learning rate $\alpha$

$w = w + \alpha \delta \nabla \hat{v}(S_t, w) $

However, in the algorithm they do not use the MSE for $\delta$, but simply

$\delta = G-\hat{v}(S_t, w)$

$\nabla\delta = - \nabla \hat{v}(S_t, w)$

which would yield a different update rule (compared to the one in the algorithm)

$w = w + \alpha \nabla \hat{v}(S_t, w) $

What am I missing?


1 Answer 1


The value $\delta$ is already representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $\hat{v}$ to get a gradient for the weights is applying the chain rule.

Your equations would work and be compatible with the book, if instead of trying to redefine $\delta$, you created the implied loss function $L = \frac{1}{2}(\hat{v}(S_t, w) - G)^2$ and set $\delta = \frac{d L}{d \hat{v}}$


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .