In the Actor-Critic example, provided by PyTorch, it seems that the update rule only occurs when the episode ends (like in a Monte-Carlo process). Specifically, in their main function they sample the environment in a loop, saving the log probabilities and state values model.saved_actions.append(SavedAction(m.log_prob(action), state_value)) right after the agent takes an action. Later, when done was reached, the loss term is aggregated and back-propagated.

The thing is - in every literature book I've come across they distinguish that one of the critical points in AC method is that the update rule can be performed after every step in the environment - something along the lines of:

  • take action $a\sim\pi_\theta(\cdot|s)$ and observe $s',r$
  • calculate TD error $\delta\gets r+\gamma \hat{V}(s',\theta_V)-\hat{V}(s,\theta_V)$
  • update $\hat{V}$'s weights $\theta_V\gets\theta_V+\alpha_V \delta \nabla_{\theta_V}\hat{V}(s,\theta_v)$
  • update $\pi_\theta$'s weights $\theta\gets\theta+\alpha_\theta I \delta \nabla_{\theta} \ln{\pi_\theta(a|s)}$

what am I missing here?


1 Answer 1


After some research (thanks, ChatGPT!) it seems that the implementation provided by PyTorch is known as Monte Carlo actor-critic, where the value function is updated using the Monte Carlo returns (i.e., the sum of the rewards received in an episode) rather than the TD error.

One advantage of Monte Carlo actor-critic is that it can be easier to implement and may be more stable in some environments. However, it is less sample efficient than the online version, since it only learns from completed episodes rather than making updates at each step.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .