I'm also a newbie in the field, so I just want to share my view.

When I was understanding this problem, I divided the problem into two scenarios. **Firstly**, the scenario where we utilize a linear policy approximator and must manually compute its gradient. **Secondly**, the case where we employ a deep policy neural network within a framework that automatically calculates the derivative.

- In the **first** case, PGT enables us to calculate the gradient without touching the derivative of stationary distribution $d^\pi(s) = \lim_{t \to \infty} P(s_t = s \vert s_0, \pi_\theta)$, which is affected by the policy and the environment. When the environment is unknown, calculating  this equation can be tricky.

- In the **second** case, PGT provides the theoretical foundation for directly optimizing the policy by computing the gradient of expected returns $J(\theta^\pi) = \mathbb{E}_{\tau \sim d^\pi(\tau)}[R(\tau)]$ ($\tau$ for trajectory). This is due to the equation:
$$
\nabla_\theta J(\theta^\pi) = \mathbb{E}_{s \sim d^\pi(s), a \sim \pi_\theta} [Q^\pi(s, a) \nabla_\theta \ln \pi_\theta(a \vert s)]
$$
It tells us that if you increase the log probability of actions that have higher expected returns (positive advantage in PPO), you can expect to increase the overall performance of the policy.

**In summary**, when you want to manually calculate the derivative of your $J$ function, PGT can be practically involved. When you use a deep learning network, PGT is also important by providing you with theoretical support, although it might not be explicitly involved.

Reference and symbol denotation: Lilian Weng's [blog](https://lilianweng.github.io/posts/2018-04-08-policy-gradient/)