2
$\begingroup$

Let

$$ \nabla_\theta J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau) \right] $$ be the expanded expression for a simple policy gradient, where $\theta$ are the parameters of the policy $\pi$, $J$ denotes the expected return function, $\tau$ is a trajectory of states and actions, $t$ is a timestep index, and $R$ gives the sum of rewards for a trajectory.

Let $\mathcal{D}$ be the set of all trajectories used for training. An estimator of the above policy gradient is given by

$$ \hat{g} = \frac{1}{\mathcal{D}} \sum_{\tau \in \mathcal{D}} \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau). $$ A loss function associated with this estimator, given a single trajectory with $T$ timesteps, is given by $$ L(\tau) = -\sum_{t = 0}^T \log \pi_\theta (a_t|s_t) R(\tau). $$ Minimizing $L(\tau)$ by SGD or a similar algorithm will result in a working policy gradient implementation.

My question is what is the proper terminology for this loss function? Is it an (unbiased?) estimator for the expected returns $J(\pi_\theta)$ if summed over all trajectories? If someone is able to provide a proof that minimizing $L$ maximizes $J(\pi_\theta)$, or point me to a reference for this, that would be greatly appreciated.

$\endgroup$
2
$\begingroup$

If I understand your question correctly, you are wondering whether the policy gradient objective coincides with some real measure of progress. This is exactly what the Policy Gradient Theorem proves (see Sutton et al. (2000) or Sutton and Barto (2018), chapter 13). In particular, policy gradient methods optimize the value of the start state $s_0$ under the current policy, $v_\pi(s_0)$. Since this value is defined as an expectation over returns, then your conclusion is correct.

Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning - an Introduction. Adaptive Computation and Machine Learning. MIT Press. http://www.worldcat.org/oclc/37293240.

Sutton, Richard S, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. “Policy Gradient Methods for Reinforcement Learning with Function Approximation.” In Advances in Neural Information Processing Systems, 1057–63.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Ah, fantastic. I'll take a look at the proof. Thanks for the references! $\endgroup$ – bxw Feb 26 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.