I have read the paper Neural Turing Machines and the paper On the Computational Power of Neural Nets about the computational power of neural networks. However, it isn't still clear to me one thing.

Is there a way of converting a neural network to another one that represents the same function? For example, given a trained recurrent neural network that represents a certain function, is it possible to convert this RNN to e.g. an MLP that represents the same function?

If an answer to the above question exists, can we show the equivalence between neural networks, e.g., all problems solved by a multilayer perceptron can be solved by a recurrent neural network, but the opposite is not true, i.e., MLP is subset of RNN (I do not know if this is true, it is just an example). So, if we obtain this relationship between all neural networks, we can get a neural network $X$ that is more powerful than others, so, we can throw away all other neural networks because $X$ can solve any problem that other NN can. Is this reasoning correct?


1 Answer 1


To answer this, it's helpful to consider the notion of a neural network architecture – in this context, we can think of the architecture as being the network depth (i.e. number of layers), width (i.e. number of nodes in a layer), and some other structural aspects, such as recurrent layers, convolution layers, pool layers, etc.


In terms of the theory, there are a few ideas that provide insight into your question. First is the commonly cited universal approximation theorem, which states that a single hidden layer, feed forward neural network with a sigmoid activation and finite width (i.e. the number of nodes in single the hidden layer) can approximate any n-dimensional real-valued function. Superficially, this tells us that we can do a whole lot with one of the most simple architectures and could (in theory) approximate more complex architectures such as LSTMs, CNNs, etc.

Yet, while this seems to indicate that reductions are always possible, research on the asymmetry in representational power between depth and width hints that things may not be so simple (for example, see this paper and this paper). Specifically, network depth provides much greater representational power relative to width. Intuitively, this makes sense – depth allows us to build hierarchies and stack multiple non-linearities. Indeed, this is where we begin to see the outlines of what the answer looks like in practice – depth provides a clear, easy way to bake structure into our network.

So, the upshot from the theoretical perspective is that the universal approximation theorem answers in the affirmative; yes, you could reduce a complex network architecture to a single hidden layer network, but the basis for doing this is a bit naïve – it doesn't consider the efficiency of the network relative to the number of parameters.The two papers linked above imply this.


In practice, there are issues with performing such a reduction. To date, the most successful models have deep architectures. More importantly though, many of these models are specialized – we have CNNs for computer vision, RNNs for sequence-based data, etc. These architectures were conceived as a way to integrate knowledge about the structure of the problem into the structure of the network to improve learning of the desired representation, and this too requires depth.

In fact, last year a paper took this idea to the extreme. Weight agnostic neural networks are specifically constructed such that their architecture is in essence the representation with a single parameter (weight) that can be varied to generate multiple distinct models with different performance characteristics. The WANNs that were developed in this research weren't especially deep, but they also weren't single layered.

The upshot here is that, for complex problems, depth is necessary for efficient learning and gradient computation and that it can be used to encode problem structure into a network. While it may be possible to reduce a learned representation to a single hidden layer network, it's not inconceivable that you may end up with more parameters condensed into a single hidden layer with very high dimensional gradients. In terms of implementation, this isn't desirable.

  • $\begingroup$ As far as I know using a complex model to train a simpler one has been done. Probably by Schmidhuber. I'm not sure they still do it. $\endgroup$ Oct 31, 2017 at 10:39
  • 1
    $\begingroup$ See also One Model To Learn Them All (2017) by Lukasz Kaiser et al. $\endgroup$ Oct 31, 2017 at 10:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .