2
$\begingroup$

Neuroevolution can be used to evolve a network's architecture (and weights, of course). Deep reinforcement learning, on the other hand, has been proven to be extremely powerful at optimising the network weights in order to train really well-performing agents. Can we use the following pipeline?

  • search for the best network topology/weights through neuroevolution
  • train the best candidate selected above through DQN or something similar

This seems reasonable to me, but I haven't found anything on the matter.

Is there any research work that attempts to combine neuroevolution with deep reinforcement learning? Is it feasible? What are the main challenges?

$\endgroup$
1
$\begingroup$

There are 3,810 articles easily found in an academic article search. These are three examples.

  • Neuroevolution for reinforcement learning using evolution strategies — C Igel — The 2003 Congress on Evolutionary Computation, 2003, ieeexplore.ieee.org — "We apply the CMA-ES, an evolution strategy which efficiently adapts the covariance matrix of the mutation distribution, to the optimization of the weights of neural networks for solving reinforcement learning problems. It turns out that the topology of the networks considerably ..."

  • Neuroevolution strategies for episodic reinforcement learning — V Heidrich-Meisner, C Igel — Journal of Algorithms, 2009, Elsevier — "Because of their convincing performance, there is a growing interest in using evolutionary algorithms for reinforcement learning. We propose learning of neural network policies by the covariance matrix adaptation evolution strategy (CMA-ES), a randomized variable-metric ..."

  • Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning — FP Such, V Madhavan, E Conti, J Lehman — 2017, arxiv.org — "Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement ..."

To clarify the strategy proposed, we can rewrite the approach as a set of design features. Let's not assume that topology formation is based on weights, which diminishes the concept of morphology and topology. If we were to reduce the model to a traditional artificial network of orthogonal topology, it would then not be neuroevolution; it would then be basic machine learning.

  • Search for the best network topology through neuroevolution
  • Train the best candidate selected above through Q-learning

The second item doesn't seem address the relationship between the inputs, outputs, and objectives of neuroevolution designs and those of Q-learning and other reinforcement learning strategies. Q-learning algorithms are not designed to run on feed forward networks in general and certainly not easily mapped to the topologies that may form during neuroevolution. There are probably billions, if not an infinite number, of ways to combine the two strategies, but simplistic concatenation of the two processes is not possible without further research and consideration of how they will interrelate collaboratively toward program goals.

It may be useful to search for articles, study, and then formulate your research trajectory. It is recommendable to start with learning about neuroevolution and reinforcement independently, and then start reading articles like the above three. Pour the foundation, let it dry, and then frame the house.

$\endgroup$
-1
$\begingroup$

Check out the NEAT algorithm (Neuro-Evolution of Augmenting Topologies). It is a way of evolving a NN through evolution. There's no reinforcement of an individual NN, but it's not needed with this algorithm. You will instead evolve the weights in the next generation.

A high level overview video: https://www.youtube.com/watch?v=qv6UVOQ0F44#action=share

The original paper: http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf

The original source code (C++) of the algorithm can be found here: https://www.cs.ucf.edu/~kstanley/neat.html

$\endgroup$
  • $\begingroup$ Hi! I appreciate your answer, but I already know about NEAT. I was specifically looking for examples of how to combine neuroevolution (such as NEAT) with DRL (such as DQN). $\endgroup$ – olinarr Jun 22 at 14:24
  • 1
    $\begingroup$ Ah sorry I don't know anything off the top my head. I wonder if going through rounds of topology evolution, weight reinforcement learning, evaluation, then topology evolution again would be good. That way you abstract the weight learning from the topology. I'll have to check back here to see if anyone posts something better. I'm curious now too. $\endgroup$ – paul_roland Jun 22 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.