Why a single trajectory can be used to update the policy network $\theta$ in A3C?

Question

The Deep RL bootcamp on policy gradient techniques gives the update equation for the policy network in A3C as

$\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}log\pi_{\theta_i}(u_t^{(k)} | s_t^{(k)})(Q(s_t^{(k)},u_t^{(k)}) - V_{\Phi_i}^\pi(s_t^{(k)})) $

However in the actual A3C paper, the gradient update is based on a single trajectory and there is no averaging of the gradient over $m$ trajectories as defined in the video ? The simple action-value actor-critic algorithm also does not seem to require an averaging over m trajectory.

Answer 1

I guess the gradient of the expectation of the Utility function, $\nabla_{\theta}J(\theta)$ in policy gradient methods where $\nabla_{\theta}J(\theta) = E_{\tau \sim p(\tau ; \theta)}[r(\tau)\nabla_{\theta}log p(\tau;\theta)]$ can be approximated using a single sample trajectory as shown in a deep reinforcement learning lecture by stanford where $J(\theta) \approx \sum_{t > 0}r(\tau)\nabla_{\theta}log\pi_{\theta}(a_t |s_t)$ and an average over sampled trajectories is not needed to compute the gradient for $\theta$ update in the direction of gradient.