What specifically is the gradient of the log of the probability in policy gradient methods?

Question

I am getting tripped up slightly by how specifically the gradient is calculated in policy gradient methods (just the intuitive understanding of it).

This Math Stack Exchange post is close, but I'm still a little confused.

In standard supervised learning, the partial derivatives can be acquired specifically because we want to learn about the derivative of the cost with respect to input parameters and then adjust in the direction of minimising this error.

Policy gradients are the opposite, as we want to maximise the likelihood of taking good actions. However, I don't understand what we are getting partial derivatives with respect to - in other words, what is the 'equivalent' of the cost function, specifically for $\nabla_\theta \log\pi_\theta$?

Answer 1

I would recommend not trying to think of this in relation to supervised learning.

The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by $\theta$. If we take a $\log$ of this function, it is still just a function. We want to take the (partial) derivative(s) of this function with respect to the parameters so that we can perform a gradient ascent step on the parameters.

A simple example can be shown by letting $\pi(a; \alpha, \beta) = \exp(\alpha + \beta a)$. In the policy gradient theorem we must first take a log of the policy which would give us $\log(\pi(a; \alpha, \beta)) = \alpha + \beta a$, and the partial derivatives wrt to the parameters are $\nabla_\alpha \log(\pi(a; \alpha, \beta)) = 1$ and $\nabla_\beta \log(\pi(a; \alpha, \beta)) = a$. We can then use these partial derivatives to perform a gradient ascent update in the direction of the gradient of our objective (the value function, which is of course what we want to maximise) for $\alpha$ and $\beta$ for a given return $G_t$ and action $a_t$ by \begin{equation} \alpha' = \alpha + G_t \times \nabla_\alpha \log(\pi(a_t; \alpha, \beta)) = \alpha + G_t \times 1 \; \\ \beta' = \beta + G_t \times \nabla_\beta \log(\pi(a_t; \alpha, \beta)) = \beta + G_t a_t\;. \end{equation}

In practice, however, you're likely to need a much more complex function for the policy, typically a neural network of some description. However, everything translates to these more complex functions, you're just going to have many more partial derivatives to calculate.

Answer 2

Consider a function $f(x)$ where $x$ is a random variable, whose distribution depends on $\theta$. The objective is to minimize \begin{align*} \mathbb{E}_x[f(x)] = \int_x f(x) \pi(x, \theta) dx \end{align*} where $\pi(x, \theta)$ is the probability density of $x$ given the parameters $\theta$ (to be formal, you should use dummy variables in place of x). The gradient is then \begin{align*} \nabla_{\theta} \mathbb{E}_x[f(x)] = \int_x f(x) \nabla_{\theta}\pi(x, \theta) dx \label{1}\tag{1} \end{align*} Eq. (1) is essentially the policy gradient. I think when written like Eq. (1), it's clear to see what the gradient actually means.

To see why this is the policy gradient, first we have \begin{align*} \int_x f(x) \nabla_{\theta}\pi(x, \theta) dx = \int_x f(x) \nabla_{\theta}\pi(x, \theta) \dfrac{\pi(x, \theta)}{\pi(x, \theta)}dx = \mathbb{E}_x[ f(x) \nabla_{\theta}\log \pi(x, \theta)]. \end{align*} Now interpret $x$ as the actions, and interpret $f(x)$ to be the expected return. I omit the full details of this, but hopefully you get the idea.

The policy gradient is simply the gradient of the expected returns, with respect to the parameters of the action distribution.