Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
In Salimans et al, 2016, the authors argue that ES should be considered a competitive alternative to MDP-based RL algorithms like Q-Learning, TRPO.
However, in practice, I notice that more often than not ES takes far more episodes to converge than MDP-based algorithms. So what would still be a reason to consider those, apart from pure academic interest?
The authors mention that ES will show less variance in long-horizon tasks, but didn't give an example. Is this aspect crucial ?
Great question! I did some research and found out that Generally capable agents emerge from open ended play by Deepmind is using ES in form of population based training:
We also explored the question, what distribution of training tasks will produce the best possible agent, especially in such a vast environment? The dynamic task generation we use allows for continual changes to the distribution of the agent’s training tasks: every task is generated to be neither too hard nor too easy, but just right for training. We then use population based training (PBT) to adjust the parameters of the dynamic task generation based on a fitness that aims to improve agents’ general capability. And finally we chain together multiple training runs so each generation of agents can bootstrap off the previous generation.
But this didn't really answer on their reasoning, so I dug some deeper and found a great article on lesswrong.com about the use of PBT. I will quote the essence, but highly recommend to read the linked Chapter on PBT:
What does the evolutionary selection give us that we don't already have? What problem does this let us avoid?
There are several answers to this question.
The more narrow answer is that this allows the dynamic task generation hyper parameters themselves to shift in a direction that promotes general competence. Neither of the optimization levels beneath us include any way of changing these parameters. But the ideal filtering parameters for the production of general competence might be different at the beginning, or at the middle of training. Or they might be different from agent to agent. Without something like population-based-training, they would have no way of changing and this would hurt performance.
The less narrow answer, I think, is that this ensures that agents are developing broad competence in a way the innermost loop cannot do. [...] each agent in our population of agents will learn to get better at some distribution of tasks, then, but without population-based-training they might not spread themselves broadly across the entire span of this distribution. Like a student who advances wildly at subjects she prefers, while ignoring subjects she is not good at, our agents might not approach our desired ideal of general competence. Population-based-training helps prevent the scenario by multiplying agent / teacher pairs that do well generally and non-narrowly.
