Is there any precedent for using a neuroevolution algorithm, like NEAT, as a way of getting to an initialization of weights for a network that can then be fine-tuned with gradient descent and back-propagation?
I wonder if this may be a faster way of getting to a global minimum before starting a decent to a local using backpropagation with a large set of input parameters.
The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and Christian W. Omlin, uses genetic algorithms to train a recurrent neural network and then uses gradient descent to fine tune the trained model.
The paper Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks (2018), by Xiaodong Cui, Wei Zhang, Zoltán Tüske and Michael Picheny, also combines evolutionary algorithms and gradient descent, but, in this case, they alternate between a gradient descent step and an evolution step. This is an example of a evolutionary stochastic gradient descent (ESGD) method, as opposed to a population-based training (PBT) method, which uses only evolutionary algorithms to train neural networks.
Yes it can be in addition to the papers that nbro linked to above uber's ai research team has a very interesting combination of sgd and neuroevolution which they have dubbed "safe mutations". In the algorithm each genome undergoes a bit of sgd to improve its fitness before the speciation, elitism, and reproduction processes. I imagine this has an effect of searching for genomes which are well suited for sgd optimization, and in my opinion does really provide the best of both a worlds. Here is the link to the paper https://arxiv.org/abs/1712.06563 . What I think would be a cool for this combination of the two would be its use in conjunction with the es-hyperneat/hyperneat neuroevolution algorithms in which a small genome cppn encodes large phenotype rnns using the rnns substrate (its structure represented with cartesian coordinates) as the cppns input. If a small amount sgd is used on the rnn's to improve fitness then what you end up with is a cppn is being evolved to encode very general rnn networks that can then be optimized to specific domains via sgd. I like this because then your neuroevolution doesnt occur on a massive rnn and you can create cppns that recognize the general problem you wish to solve if your clever with your fitness evaluation.
