# Can we simply remove the log term for loss in policy gradient methods?

If I understand correctly, the goal of vanilla policy gradients is maximizing $$E[r(s_t,a_t);\pi_\theta]$$; in deriving the gradient of this function as a clearer function on $$\theta$$, we get $$\sum_{t=0}^{T-1}\nabla_\theta \log\pi_\theta(a_t|s_t)G_t$$: a sort of weighted sum of the gradients of log probabilities; we treat $$\log\pi_\theta(a_t|s_t)G_t$$ as basically equivalent to $$E[r(s_t,a_t);\pi_\theta]$$ as their gradients are essentially equal: maximizing one maximizes the other. In PPO, we arbitrarily replace this with $$E[\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{prev}}(a_t|s_t)}A_t]$$ + clipping. It seems that as long as higher rewards are associated with higher probabilities, anything in the form $$E[\pi_\theta A_t]$$ is fine to maximize and will also maximize $$E[r(s_t,a_t);\pi_\theta]$$. Is this correct? Can I simply remove the log from the equation, for example?

• since $\log$ is a strictly monotonic function, you probably can remove the this, but 1) it is not necessarily theoretically motivated, 2) it will almost certainly lead to instabilities. That is why we generally talk about log-likelihoods in stats/ML, as it is much easier to optimiser from a numerical stability point of view. Commented Jul 28 at 21:14

Let's say we have two possible actions $$a_1,a_2$$ of a simple bandit problem with the parameterized policy for both actions defined as $$\pi_{\theta}(a_1)=\sigma(\theta), \pi_{\theta}(a_2)=1-\sigma(\theta)$$, where $$\sigma$$ is the standard sigmoid function. Also suppose the rewards for the actions are $$r(a_1)=1, r(a_2)=0$$, since it's a bandit without state the return $$G_t$$ is simply $$r(a)$$ for each action $$a$$. Then it's straightforward to compute gradient as follows: $$\nabla_{\theta} J(\theta)=\mathbb{E}_{\pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(a)G_t]=\pi_{\theta}(a_1)(1-\sigma(\theta))r(a_1)+\pi_{\theta}(a_2)(-\sigma(\theta))r(a_2)=\sigma(\theta)(1-\sigma(\theta))$$ Similarly you can get the gradient for the case of removing the log as $$\nabla_{\theta} J'(\theta)=\sigma(\theta)^2(1-\sigma(\theta))$$
Therefore clearly even in 1-d bandit case the gradient without log will become extremely small if $$\sigma(\theta)$$, i.e., $$\pi_{\theta}(a_1)$$ is very small, which means the initially very unlikely actions in the to-be-optimized policy cannot get updated efficiently consistent with above intuition due to losing the proper scaling effect of the log. And in the usual multi-dimensional policy for MDP cases both the magnitude and direction of the gradient will be similarly negatively impacted.
Finally with the common log gradient trick the existing form in PGT can be simply transformed to a quotient where the numerator is just your gradient of the policy without log and the denominator is the same policy. From the balance need of exploitation-exploration, this denominator is required for exploration since if the probability of taking certain action in a state is small, then the algorithm would update the parameters so that the probability of taking that action can increase. The other term $$G_t \approx Q_t(s_t,a_t)$$ reflects that if an action value is great then the algorithm intends to update the parameters so that the probability of taking that action is enhanced which is exploitation. Therefore if you remove the denominator policy then equivalently you're not exploring enough.