I'm struggling a little with understanding the OpenAI implementation of A2C in the baselines (version 2.9.0) package. From my understanding, one step_model acts in different parallel environments and gathers experiences (calculates the gradients, I think), and sends them to the train_model that trains with them. After this, the step_model gets updated from the train_model.
What I am unsure about is if both step_model and train_model are actor-critic models or if step_model is actor and train_model is a critic (or vice versa). Does the step_model use the advantage function or is it just the train_model?
