Timeline for What is the difference between A2C and running an agent in an environment vector?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 3, 2020 at 2:36 | comment | added | jgauth | @NeilSlater Awesome explanation Neil! | |
| Apr 4, 2019 at 6:45 | comment | added | Neil Slater | @SherwinChen: Clearly I am not able to explain this in your terms in the comments. Consider asking another question on the site | |
| Apr 3, 2019 at 23:57 | comment | added | Maybe | Hi @NeilSlater, and thanks again. I'm still quite confused about why, in the case of using environment vector, we can predict the next observation to train from in the way that we cannot do the same to multiple agents. In A2C, even we use multiple agents, they are just nothing but copies of the learning agent since every time we update the learning agent, we synchronize the updated parameters with all agents. Environments are different in both cases since we use different seeds. What on earth makes the difference? Or am I misunderstanding something? | |
| Apr 3, 2019 at 9:20 | comment | added | Neil Slater | @SherwinChen: A single agent's vectors will be correlated. E.g. you can predict the next observation to train from with some degree of accuracy better than random, even knowing nothing about the agent. Whilst if you take the next observation from a different agent, your ability to predict what the state is in advance is reduced. Self-correlated data is not i.i.d. and you want to avoid that, especially when training incrementally using online learners such as neural networks | |
| Apr 3, 2019 at 7:05 | comment | added | Maybe | @NeilSlater, thanks for responding, but I'm sorry that I cannot really understand what you were trying to convey at the end: "In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources." I've updated my question to include a simple environment vector, I cannot see why the data will be less i.i.d. than that collected by multiple agents. I'm looking for your response | |
| Apr 3, 2019 at 6:43 | comment | added | Neil Slater | @SherwinChen: There is no difference between "multiple agents" and "multiple environments" by default, unless you are deliberately connecting the agents in the former (e.g. forcing them to the same state on each time step) or deliberately changing the environments in the latter (e.g. changing the state transition rules). Neither of those things are suggested here. The point is to reduce correlation. In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources. | |
| Apr 3, 2019 at 1:38 | comment | added | Maybe | Hi, I still feel confused. In general, A2C is implemented with an on-policy policy-gradient algorithm, If we use different parameters for different agents, then the learning is not on-policy at all. Why would we do that? BTW, @NeilSlater, I cannot see how multiple agents reduce correlation in the way multiple environments do not. Could you please shed some light on it? | |
| Mar 31, 2019 at 11:40 | comment | added | Neil Slater | In addition, the multiple agents approach is used for similar reasons to DQN's experience replay - it helps to reduce correlation within training batches. This can be important for stability of a neural network within the A2C framework. | |
| Mar 31, 2019 at 9:00 | vote | accept | Maybe | ||
| Apr 3, 2019 at 7:00 | |||||
| Mar 31, 2019 at 8:09 | history | answered | Brale | CC BY-SA 4.0 |