How is neural architecture search performed?

Question

I have come across something that IBM offers called neural architecture search. You feed it a data set and it outputs an initial neural architecture that you can train.

How is neural architecture search (NAS) performed? Do they use heuristics, or is this meta machine learning?

If you have any papers on NAS, I would appreciate if you can provide a link to them.

Answer 1

You could say that NAS fits into the domain of Meta Learning or Meta Machine learning.

I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very interesting. It's sorted in rough chronological descending order, and *** means influential / must read.

Quoc V. Le and Barret Zoph are to good authors on the topic.

• The Evolved Transformer
• Exploring Randomly Wired Neural Networks for Image Recognition
• GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURE SEARCH
• Backprop Evolution
• Progressive Neural Architecture Search
• *** DARTS: Differentiable Architecture Search
• *** Efficient Neural Architecture Search via Parameter Sharing - ENAS
• *** Progressive Neural Architecture Search
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Automatic Machine Learning - Prof. Frank Hutter
• Google Brain - Neural Architecture Search - Quoc Le
• *** Regularized Evolution for Image Classifier Architecture Search
• Autostacker: A Compositional Evolutionary Learning System
• Generating Neural Networks with Neural Networks
• Finding Competitive Network Architectures Within a Day Using UCT
• Neuroevolution: A different kind of deep learning
• Evolving Deep Neural Networks
• Pieter Abbeel: Deep Learning-to-Learn Robotic Control
• *** SMASH: One-Shot Model Architecture Search through HyperNetworks

Answer 2

Here are two review articles:

• Elsken, Metzen, Hutter: Neural Architecture Search: A Survey (2019), Journal of Machine Learning Research 20, 1-21

• He, Zhao, Chu: AutoML: A Survey of the State-of-the-Art (2019)

Answer 3

Neural architecture search (NAS) is a method of automating the design (that is, the choice of the values of the hyper-parameters) of artificial neural networks. There are different approaches to search the space of neural network architectures. For example, you can use reinforcement learning or evolutionary (or genetic) algorithms.

Check out the paper Neural Architecture Search with Reinforcement Learning (2017), by Barret Zoph and Quoc V. Le, where the authors train, using reinforcement learning (specifically, REINFORCE), a recurrent neural network (the "controller") to generate (convolutional and recurrent) neural network architectures, so that to maximise the expected accuracy of the generated architectures on a validation dataset. They achieve some good results using this approach.

See also Efficient Neural Architecture Search via Parameter Sharing (2018), by Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le and Jeff Dean (which thus includes some of the authors of NAS), which is similar to NAS, but more efficient, hence the acronym ENAS (efficient NAS).