Is the negative of the policy loss function in a simple policy gradient algorithm an estimator of expected returns?

Question

Let

$$ \nabla_\theta J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau) \right] $$ be the expanded expression for a simple policy gradient, where $\theta$ are the parameters of the policy $\pi$, $J$ denotes the expected return function, $\tau$ is a trajectory of states and actions, $t$ is a timestep index, and $R$ gives the sum of rewards for a trajectory.

Let $\mathcal{D}$ be the set of all trajectories used for training. An estimator of the above policy gradient is given by

$$ \hat{g} = \frac{1}{\mathcal{D}} \sum_{\tau \in \mathcal{D}} \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau). $$ A loss function associated with this estimator, given a single trajectory with $T$ timesteps, is given by $$ L(\tau) = -\sum_{t = 0}^T \log \pi_\theta (a_t|s_t) R(\tau). $$ Minimizing $L(\tau)$ by SGD or a similar algorithm will result in a working policy gradient implementation.

My question is what is the proper terminology for this loss function? Is it an (unbiased?) estimator for the expected returns $J(\pi_\theta)$ if summed over all trajectories? If someone is able to provide a proof that minimizing $L$ maximizes $J(\pi_\theta)$, or point me to a reference for this, that would be greatly appreciated.

Answer 1

If I understand your question correctly, you are wondering whether the policy gradient objective coincides with some real measure of progress. This is exactly what the Policy Gradient Theorem proves (see Sutton et al. (2000) or Sutton and Barto (2018), chapter 13). In particular, policy gradient methods optimize the value of the start state $s_0$ under the current policy, $v_\pi(s_0)$. Since this value is defined as an expectation over returns, then your conclusion is correct.

Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning - an Introduction. Adaptive Computation and Machine Learning. MIT Press. http://www.worldcat.org/oclc/37293240.

Sutton, Richard S, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. “Policy Gradient Methods for Reinforcement Learning with Function Approximation.” In Advances in Neural Information Processing Systems, 1057–63.