Is the policy gradient expression in Fundamentals of Deep Learning wrong?

Question

I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning.

Here is the whole paragraph:

Policy Learning via Policy Gradients

In typical supervised learning, we can use stochastic gradient descent to update our parameters to minimize the loss computed from our network's output and the true label. We are optimizing the expression: $$ \arg \min _{\theta} \Sigma_{i} \log p\left(y_{i} \mid x_{i} ; \theta\right) $$ In reinforcement learning, we don't have a true label, only reward signals. However, we can still use SGD to optimize our weights using something called policy gradients. We can use the actions the agent takes, and the returns associated with those actions, to encourage our model weights to take good actions that lead to high reward, and to avoid bad ones that lead to low reward. The expression we optimize for is: $$ \arg \min _{\theta}-\sum_{i} R_{i} \log p\left(y_{i} \mid x_{i} ; \theta\right) $$ where $y_{i}$ is the action taken by the agent at time step $t$ and where $R_{i}$ is our discounted future return. A In this way, we scale our loss by the value of our return, so if the model chose an action that led to negative return, this would lead to greater loss. Furthermore, if the model is very confident in that bad decision, it would get penalized even more, since we are taking into account the log probability of the model choosing that action. With our loss function defined, we can apply SGD to minimize our loss and learn a good policy.

The first expression about the loss computed in a network already seems false since the log of a probability is always negative, and taking the $θ$ (weights) for which the expression is minimal doesn't seem right because it would favor very unsure answers.

The same goes with the next expression on policy gradient. A very negative $R_i$ and very unsure $p(y_i)$ would both be big negatives and multiplied together give a big positive value. Since there is a - sign in front of the expression, this would be the best configuration for the argmin. Meaning we are looking for weights in the policy that give highly negative rewards and for highly unsure actions. This just doesn't make sense to me.

Is it just a sign error (or we could just change to argmax)? Or is there more to it?

Answer 1

There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.

In short, we want to first find the derivative of the RL objective $J(\theta) = v_\pi(s_0)$, where $\pi$ is our policy that depends on some parameters $\theta$. The policy gradient theorem tells us that $$\nabla_\theta J(\theta) = \mathbb{E}_{a\sim\pi, s\sim\mu}\left[ G_t \nabla_\theta \log\pi(A_t|S_t)\right]\;;$$ where $\mu$ is our state distribution induced by the policy $\pi$ and $G_t$ are the discounted returns.

Now, as we want to maximise our objective $J(\theta)$, we want to perform gradient ascent. That is, our parameters should be updated according to $$\theta_{t+1} = \theta_t + \alpha \nabla_\theta J(\theta)\;.$$

The reason that you will often see the update written as a minimisation problem is because most software is built to minimise functions, rather than maximise. This is not a problem though as finding the maximum of $f$ is just the same as finding the minimum of $-f$. Using this, we can then say our parameters can be updated using gradient descent on the negative of our objective, i.e. we get our parameter update rule to be $$\theta_{t+1} = \theta_t - \alpha \nabla_\theta (-J(\theta))\;;$$ which is exactly what you have in the book.