Let
$$ \nabla_\theta J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau) \right] $$ be the expanded expression for a simple policy gradient, where $\theta$ are the parameters of the policy $\pi$, $J$ denotes the expected return function, $\tau$ is a trajectory of states and actions, $t$ is a timestep index, and $R$ gives the sum of rewards for a trajectory.
Let $\mathcal{D}$ be the set of all trajectories used for training. An estimator of the above policy gradient is given by
$$ \hat{g} = \frac{1}{\mathcal{D}} \sum_{\tau \in \mathcal{D}} \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau). $$ A loss function associated with this estimator, given a single trajectory with $T$ timesteps, is given by $$ L(\tau) = -\sum_{t = 0}^T \log \pi_\theta (a_t|s_t) R(\tau). $$ Minimizing $L(\tau)$ by SGD or a similar algorithm will result in a working policy gradient implementation.
My question is what is the proper terminology for this loss function? Is it an (unbiased?) estimator for the expected returns $J(\pi_\theta)$ if summed over all trajectories? If someone is able to provide a proof that minimizing $L$ maximizes $J(\pi_\theta)$, or point me to a reference for this, that would be greatly appreciated.