Taking as an example the Advantage Actor Critic, the objective function is: \begin{equation} \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\sum_{t=0}^T \nabla_{\boldsymbol{\theta}} \log \pi_{\boldsymbol{\theta}}\left(a_t \mid s_t\right)A^{\pi, \gamma}(s_t, a_t)\right], \end{equation} where $A^{\pi, \gamma}(s_t, a_t) = Q^{\pi, \gamma}(s_t, a_t) - V^{\pi, \gamma}(s_t)$ is the advantage function.

In principle in order to compute the gradient we should collect many trajectories and take the average of the desired quantity (which is what REINFORCE does). I guess that computing many trajectory is very costly, and indeed in Actor-Critic-like algorithms, such gradient is estimated by sampling just one trajectory (the policy is updated after every episode).

I understand that we exploit baselines in order to reduce variance for better estimates but I would like to understand why the one-trajectory sample is a good estimator of $\nabla_{\theta}J(\theta)$. Are there any rigorous references about this?