Why is policy gradient theorem so important?

Question

What is the problem that the policy gradient solves? From what I understand the problem is taking the gradient of the state distirbution $d^{\pi_{\theta}}$, but what is exactly the problem here (maybe it is a practical problem more than a theoretical one)? After applying the PGT we don't have this issue anymore. Thanks for any explanation!

Answer 1

I'm also a newbie in the field, so I just want to share my view.

When I was understanding this problem, I divided the problem into two scenarios. Firstly, the scenario where we utilize a linear policy approximator and must manually compute its gradient. Secondly, the case where we employ a deep policy neural network within a framework that automatically calculates the derivative.

• In the first case, PGT enables us to calculate the gradient without touching the derivative of stationary distribution $d^\pi(s) = \lim_{t \to \infty} P(s_t = s \vert s_0, \pi_\theta)$, which is affected by the policy and the environment. When the environment is unknown, calculating the derivative of this equation can be tricky.

• In the second case, originally, the objective function should be $J(\theta^\pi) = \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)]$, which can be used to evaluate the performance of the policy $\pi_\theta$. The PGT tells us that the gradient of this equation is $$ \nabla_\theta J(\theta^\pi) = \mathbb{E}_{s \sim d^\pi(s), a \sim \pi_\theta} [Q^\pi(s, a) \nabla_\theta \ln \pi_\theta(a \vert s)] $$ Therefore, we leverage on that to build the loss function we used to train the neural network as: $$ \mathbb{L}(\theta^\pi) = - \mathbb{E}_{s \sim d^\pi(s), a \sim \pi_\theta} [Q^\pi(s, a) \ln \pi_\theta(a \vert s)] $$ Note that this loss function does not approximate $J(\theta^\pi) = \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)]$. This loss function is only useful because it has a negative gradient of the performance when evaluated at the current parameters. "This means that minimizing this 'loss' function, for a given batch of data, has no guarantee whatsoever of improving expected return." (Ref: Spinning Up Doc)

In summary, when you want to manually calculate the derivative of your $J$ function, PGT can be practically involved. When you use a deep learning network, PGT is also important by providing you with theoretical support of how to construct your loss function.

Reference and symbol denotation:

1. Lilian Weng's blog
2. Spinning Up Doc