When we should use NEAT and GA? Today the problem of “We do not know which answer is correct” was solved by Reinforcement Learning, so when using genetic algorithm is still useful, and do we have RL that changes topology of NN?
2
-
1$\begingroup$ 'Today the problem of “We do not know which answer is correct” was solved by Reinforcement Learning' - What? That's a very unclear way of phrasing what RL does. Can you clarify what you mean by "why when using genetic algorithm is still useful"? Why or when? It's unclear. This post is so confusing and it's unclear why you're asking the question in the title, which is the only thing that is more or less clear. $\endgroup$– nbroCommented Dec 5 at 11:01
-
$\begingroup$ That is why = so $\endgroup$– Nikolai VorobievCommented Dec 5 at 11:33
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Generally speaking, when architecture search is essential or the problem objective is non-differentiable while creativity and diversity in solutions matter most, use NEAT/GA. When rewards are well-defined, environments are complex but problem objective is differentiable, and ML model topology is fixed, use RL.
Having said that, there're RL approaches changing the topology of neural networks during training. One popular framework is NAS/AutoML which employs RL to adapt and optimize ANN topology.
Reinforcement learning (RL) can underpin a NAS search strategy. Barret Zoph and Quoc Viet Le applied NAS with RL targeting the CIFAR-10 dataset and achieved a network architecture that rivals the best manually-designed architecture for accuracy... In the so-called Efficient Neural Architecture Search (ENAS), a controller discovers architectures by learning to search for an optimal subgraph within a large graph. The controller is trained with policy gradient to select a subgraph that maximizes the validation set's expected reward.
In some cases NAS variant also employs evolutionary algorithms including GA to adapt and optimize ANN topology.
An Evolutionary Algorithm for Neural Architecture Search generally performs the following procedure... On CIFAR-10 and ImageNet, evolution and RL performed comparably, while both slightly outperformed random search.
-
$\begingroup$ To be sure, by GA we mean crossing and changing the weights of neural networks while leaving the topology unchanged? $\endgroup$ Commented Dec 5 at 10:46
-
$\begingroup$ Not necessarily, here meant for your concerned hyperparameters optimization of topology of ANN, see GA definition: Genetic algorithms are commonly used to generate high-quality solutions to optimization and search problems via biologically inspired operators such as selection, crossover, and mutation. Some examples of GA applications include optimizing decision trees for better performance, solving sudoku puzzles, hyperparameter optimization, and causal inference. Hope this clarifies your lingering concern and helpful for your question. $\endgroup$– cinchCommented Dec 5 at 18:41
-
$\begingroup$ Thank you, I understand that GA is a more general concept that includes NEAT? And I can't remember when a problem is not differentiable for NN, give a couple of examples $\endgroup$ Commented Dec 5 at 20:07
-
$\begingroup$ If you only restricted to NN with non-differentiable objective, my above answer deals with hyperparameter optimization of ANN topologies for your OP since number of layers, types of activations, adding/removing nodes are all non-differentiable. Also even in RL such as AlphaZero, you still need MCTS heuristics where there's no gradient to backprop. And your loss in some cases could be F1 or BLEU (for language generation) liek non-differentiable functions. Hope this clarifies your lingering question, if you have more related questions, you could post new question. $\endgroup$– cinchCommented Dec 5 at 21:28
-
$\begingroup$ so RL uses gradient descent and for this the task should be differentiable, however, as You mentioned we have AutoML, so now I am confused - if RL requires differentiability, but also, as you said, there are solutions for non-differentiable questions with reinforcement learning, I am completely lost in the algorithm's operation $\endgroup$ Commented Dec 7 at 7:48