# Tag Info

12

Section 4.2 of "Essentials of Metaheuristics" has a wealth of information on alternative ways of encoding graph structures via Genetic Algorithms. With particular regard to evolving ANNs, I would personally not be inclined to implement this sort of thing 'from scratch': The field of neuroevolution has been around for some time, and the implementation some ...

11

I think in this case, you'll probably want to use a genetic algorithm to generate a topology rather than working on your own. I personally like NEAT (NeuroEvolution of Augmenting Topologies). The original NEAT paper involves evolving weights for connections, but if you only want a topology, you can use a weighting algorithm instead. You can also mix ...

7

To find the number of neurons and layers that you will use is not that straightforward. The best way to do this is through experimentation however you will be able to better estimate the number of layers and neurons needed through experience. One of the common rules is that more neurons are better for more complex datasets. However, you do not want too many ...

7

When crossover happens and one parent is fitter than the other, the nodes from the more fit parent are carried over to the child. This is the case as disjoint and excess genes are only carried over from the fittest parent. Here's an example: // Node Crossover Parent 1 Nodes: {[0][1][2]} // more fit parent Parent 2 Nodes: {[0][1][2][3]} Child Nodes: {[0]...

6

You could say that NAS fits into the domain of Meta Learning or Meta Machine learning. I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very interesting. It's sorted in rough chronological descending order, and *** means influential / must read. Quoc V. Le and Barret Zoph are to good authors on the ...

5

Yes, this is an active area of research as we speak. Both using classic algorithms (decision trees, random forests, Bayesian ensembles) as well as neural networks. This can also be done via evolutionary algorithms. I have personally used them for hyperparameter tuning in a few cases where squeezing out a couple of extra points of accuracy was key. This is ...

5

The other answer mentions NEAT to generate network weights or topologies. The paper NeuroEvolution: The Importance of Transfer Function Evolution and Heterogeneous Networks, which also gives a short summary of neuroevolution techniques, provides an alternative approach to NEAT. It uses Cartesian Genetic Programming to evolve a multiple activation functions.

5

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves ...

4

Using evolutionary algorithms to evolve neural networks is called neuroevolution. Some neuroevolution algorithms optimize only the weights of a neural network with fixed topology. That sounds not like what you want. Other neuroevolution algorithms optimize both the weights and topology of a neural net. These kinds of algorithms seem more appropriate for ...

4

NEAT has a constant number of organisms in its population, which prevents overpopulation from happening. The process of mating includes the following steps. The worst networks from every species are removed. All species receive a number of offsprings that they can have. This is calculated by an adjusted neural network fitness. Offsprings for species are ...

4

It is actually the other way around: connection IDs is what is debated! Nodes always have innovation IDs (in the image, it is just their identifying number). Node IDs are sufficient to identify connections. If a connection links nodes 3 and 6, then it is the same as another connection linking nodes 3 and 6: no need for an extra ID. So why the extra ...

4

Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it ...

4

The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and Christian W. Omlin, uses genetic algorithms to train a recurrent neural network and then uses gradient descent to fine tune the trained model. The paper Evolutionary ...

4

The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers building on or using the algorithm. NEAT is not widely used in commercial applications because if you have a clean objective function, a topology that is optimized ...

3

What is the definition of "structural innovation", and how do I store these so I can check if an innovation has already happened before? Structural innovation is anything added that changes the topology of the network. So a structural innovation is any added connection or added node. I don't want to get too much into the implementation, but something ...

3

NEAT uses genetic algorithms both to search for improved connection weights and for improved architectures. Whilst it is possible to train a NEAT-generated neural network using backpropagation of error gradients, libraries implementing "original" NEAT will not implement that. There are a couple of reasons: There is often no training data, in a supervised ...

3

Yes, the original gene is disabled, but is left in the genome. This can be seen on page 10, figure 3 of the paper linked (taken from the original paper NEAT Paper) where gene 3 is disabled, but not removed from the genome. This gene can be re-enabled by receiving the gene with the identical innovation number from a mating partner with the gene enabled during ...

3

This answer points at some of the more modern approaches. This has been around for a long time in the form of NeAT: Neuroevolution of Augmenting Topologies, originally described in Kenneth Stanley's 2002 paper. NeAT is available as a package for many languages, including Python, Java, and C++. The algorithm works as a form of genetic programming. A ...

3

It is possible, but is a pretty terrible idea. There are a few options. One is to not use the GA as a direct classifier, but instead use a GA to learn the parameters of another classification model like a neural network. The basic idea of a GA is that it (very roughly speaking) forms a black-box method for searching an arbitrary space for solutions that ...

3

In the paper Exploiting Open-Endedness to Solve Problems Through the Search for Novelty (2008), by Joel Lehman and Kenneth O. Stanley, which introduced the novelty search approach, it is written Thus this paper introduces the novelty search algorithm, which searches with no objective other than continually finding novel behaviors in the search space. and ...

3

I would first say consider the advice of Thomas W in the comment above and think about whether you really need to discretize your variables. I'd also question the wisdom of training a reasonably sized network with a GA instead of a dedicated neural net training algorithm that's very likely to exhibit much better performance. However, assuming you really do ...

3

When does the mutation occur and how does it take place? Finding a solution in NEAT algorithm is based on evolution strategy. It means that you have Neural Networks which are yours individuals, so mutations and crossing occurs in loop after phase of "fitnessing" (calculation fitness for every individual and removing bad ones). How is it chosen whether ...

3

I agree with Aiden Grossman that perhaps you should try another algorithm before messing with NEAT, as NEAT is fairly complex. However, I thought I might explain the benefits of NEAT as they pertain to the second part of your question. The NEAT method is from a paper written by Kenneth O. Stanley and Risto Miikkulainen titled Evolving Neural Networks ...

2

When do mutations occur and between which nodes? There are two types of mutations in the NEAT model, each of them appears randomly during one epoch on different individuals; the number of structures affected by mutations may vary depending upon the nature of the problem. A new gene/node is added to the structure and properly linked. A new connection ...

2

The GA will require a fitness function, which means you need labeled data for comparison. That conclusion is wrong. Yes, sometimes your fitness function will use labeled data. For example, if you want to train an XOR gate or any other known function. However, there is arguably no advantage of training a function with neuroevolution versus backpropagation, ...

2

Neural Network equivalents that is not (vanilla) feed forward Neural Nets: Neural net structures such as Recurrent Neural Nets (RNNs) and Convolutional Neural Nets (CNNs), and different architectures within those are good examples. Examples of different architectures within RNNs would would be: Long Short Term Memory (LSTM) or Gated Recurrent Unit (GRU). ...

2

You must understand that a genetic algorithm is an optimization algorithm. You can't feed it e-mails and make it classify spam. A genetic algorithm is used to train a model to classify spam. That something could be neural networks. What you need is a genetic algorithm that optimizes neural networks neuroevolution, which might roughly work as follows Start ...

2

$$1-\sum_i(e_i-a_i)^2$$ $\sum$ - there just means sum. It is the greek letter for S. You can rewrite the above formula as $$1 -[(e_1 - a_1)^2+(e_2-a_2)^2+(e_3-a_3)^2+\ldots ]$$ $\sum$ just helps us avoid writing dozens of $+$ signs. Read more here. What they are doing here is taking the difference of expected value $e_1$ and the actual value $a_1$ for the ...

2

Neil Slater is correct when saying that NEAT itself is not neural networks evolving neural networks, what I believe is the closest framework to what the question is asking would be HyperNEAT http://axon.cs.byu.edu/~dan/778/papers/NeuroEvolution/stanley3**.pdf HyperNEAT operates in a very similar way to what you are describing, from a ten thousand foot view ...

2