I have read (here and here ) about the computational power of neural networks and a doubt came up.

There is a way to reduce an ANN to another ANN (not taking into count the training algorithm) ? e.g. Reduce a Recurrent Neural Network to a Multilayer Perceptron, meaning that if I have a trained RNN, I can get a MP that maps the same inputs given to the RNN to the same outputs produced by the RNN.

And if exists an answer to the above question, we can show the equivalence between neural networks, e.g., all problems solved by an Multilayer Perceptron can be solved by a Recurrent Neural Network but the opposite is not true, i.e., MP is subset of RNN (I do not know if this is true, is just an example). So, if we obtain this relationship between all neural networks, we can get a neural network X that is more powerful than others, so, we can throw away all other neural networks because X can solve any problem that other NN can. Is this reasoning correct ?



It’s an interesting idea to train a complex model and then use it to train a simpler model that can mimic it. I can’t say for sure whether this is practiced or not, but my guess is that it isn’t; I’m basing this on the intersection of the universal approximation theorem and decades of empirical results across the field.

While there are several versions of the theory, the gist is that a feedforward neural network can be made to approximate any continuous function. This property of neural network was first shown in one of the most basic architectures — single layer, sigmoid activation, with a finite set of neurons.

Of course, we know from research in the space that this is not quite true in practice. Specifically, the architecture and parameterization of a neural network strongly influence its ability to learn certain functions — and we know that certain tasks are best approached with specific classes of architecture (e.g. conv nets for object detection).

So circling back, while a more complex architecture may be reducible to one which theoretically could learn the same function, in practice, for many tasks this is unlikely to be the case. Could we create a neural network architecture that rules them all? Maybe, but it has yet to be done in any clear way.

  • $\begingroup$ As far as I know using a complex model to train a simpler one has been done. Probably by Schmidhuber. I'm not sure they still do it. $\endgroup$ – BlindKungFuMaster Oct 31 '17 at 10:39
  • 1
    $\begingroup$ Also: One model to learn them all: arxiv.org/abs/1706.05137 $\endgroup$ – BlindKungFuMaster Oct 31 '17 at 10:45
  • $\begingroup$ Nice, I'll have to read that paper – looks interesting. $\endgroup$ – Greenstick Oct 31 '17 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.