Existing ANNs are so good for solving complicated tasks on a different domain of data. But creating a neural network is always a hassle and its success mostly relies on an engineer's intuition and skills. Most of the NN are topologically straightforward, but some of the successful ideas like Convolutional, LSTM, GRU, Attention cells are more complicated. In this sense, I believe that the LSTM cell is a good example of luck - how a human (even so smart as its author) can create a complicated design using its limited knowledge of the world of computation and describe its physical sense (input/output/forget gates).
Neuroevolution algorithms showed us, in some sense, a new way of how to create models. GoogLeNet showed us that neural networks can be much more complicated and work even without a need for our physical interpretation of its structure. And that's very exciting what neural networks we will see in the near future.
But thinking outside of the recent neural network models - this is all special cases of computational graphs, limited with some ideas of simplified neurons models (in this case McCulloch-Pitts model) or more a bit more complicated convolutional and recurrent neuron models.
But what are the recent advances in this field of computational graphs? How evolutionary algorithms (or other derivative-free optimization algorithms) can be applied to computational graphs for existing tasks to discover new ideas for neural network cells and topologies? For example, what result you will get if you apply evolutionary algorithms on a pure computational graph to optimize it for some simple tasks like MNIST classification?
Papers, thoughts, critics are welcome.