I'm struggling a little with understanding the OpenAI implementation of A2C in the baselines
(version 2.9.0) package. From my understanding, one step_model
acts in different parallel environments and gathers experiences (calculates the gradients, I think), and sends them to the train_model
that trains with them. After this, the step_model
gets updated from the train_model
.
What I am unsure about is if both step_model
and train_model
are actor-critic models or if step_model
is actor and train_model
is a critic (or vice versa). Does the step_model
use the advantage function or is it just the train_model
?