NEAT is an evolutionary algorithm. When would you want to use NEAT over more traditional/common RL algorithms like PPO or SAC etc. What advantage does it give you?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ It seems that you know what NEAT can be used for. Do you know what RL is used for? If yes, can you give an example of when you think you could use NEAT rather than RL (which doesn't mean it is necessarily a better approach than RL)? $\endgroup$– nbroCommented Aug 13, 2021 at 14:31
-
$\begingroup$ Sure, in the NEAT python repo they have examples of applying NEAT to OpenAI lunar-lander and cartpole problems. Are you asking a rhetorical question (actually just asking so i dont sound dumb lol)? $\endgroup$– Dylan KerlerCommented Aug 13, 2021 at 17:10
-
$\begingroup$ I was just making sure that you know what RL is used for, and if you know how NEAT could be applied to RL problems. Maybe you can briefly state in your post what NEAT can be used for. To find a policy? To evolve neural networks that represent the policy that you want to find in RL? $\endgroup$– nbroCommented Aug 13, 2021 at 20:16
-
1$\begingroup$ Yes, I'm very familiar with the de-facto RL like using PPO, Q-Learning etc. NEAT can be used to find a policy through "evolution" of both the neural net architecture and the weights in the neural net. I'm wondering in what situation would NEAT be better than policy gradient RL. $\endgroup$– Dylan KerlerCommented Aug 13, 2021 at 21:31
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In my opinion, this shouldn't be an either/or question. Both NEAT and rl with fixed network topology has their own advantage when solving decision problem.
NEAT is good to solve a simple problem fast with minimum network topology and without local optimum issue. While RL with fixed topology suffers local optimum but learning more directly, and policy-gradient based Rl would be even more adaptive to an environment with stochasticity in the context of statistics.
Then, why not just combine them? At present, the best paper on this topic is this： https://dl.acm.org/doi/10.1145/3205455.3205536 In this paper, it is suggested that doing NEAT at first, then KEEP DOING RL forever, though I don't agree with this special routine. I think, both NEAT and RL should do interactively during the WHOLE training process.
The problem is, how to combine these 2 in an effective way. One problem I met is for RL, like SAC, which has 2 outputs(one for policy and one for Q-value), And the topology of the Q-value output has no contribution when doing NEAT. Then, how to deal with the Q-value topology? If both the 2 outputs shares some layers at first, then the learning would become more unstable, since doing NEAT would dramatically change these 2 outputs.
The paper above takes a fixed output layer for both 2 outputs, which I think is just a workaround, not the best way.
