5
$\begingroup$

I have read the paper Neural Turing Machines and the paper On the Computational Power of Neural Nets about the computational power of neural networks. However, it isn't still clear to me one thing.

Is there a way of converting a neural network to another one that represents the same function? For example, given a trained recurrent neural network that represents a certain function, is it possible to convert this RNN to e.g. an MLP that represents the same function?

If an answer to the above question exists, can we show the equivalence between neural networks, e.g., all problems solved by a multilayer perceptron can be solved by a recurrent neural network, but the opposite is not true, i.e., MLP is subset of RNN (I do not know if this is true, it is just an example). So, if we obtain this relationship between all neural networks, we can get a neural network $X$ that is more powerful than others, so, we can throw away all other neural networks because $X$ can solve any problem that other NN can. Is this reasoning correct?

$\endgroup$
2
$\begingroup$

To answer this, it's helpful to consider the notion of a neural network architecture – in this context, we can think of the architecture as being the network depth (i.e. number of layers), width (i.e. number of nodes in a layer), and some other structural aspects, such as recurrent layers, convolution layers, pool layers, etc.

Theory

In terms of the theory, there are a few ideas that provide insight into your question. First is the commonly cited universal approximation theorem, which states that a single hidden layer, feed forward neural network with a sigmoid activation and finite width (i.e. the number of nodes in single the hidden layer) can approximate any n-dimensional real-valued function. Superficially, this tells us that we can do a whole lot with one of the most simple architectures and could (in theory) approximate more complex architectures such as LSTMs, CNNs, etc.

Yet, while this seems to indicate that reductions are always possible, research on the asymmetry in representational power between depth and width hints that things may not be so simple (for example, see this paper and this paper). Specifically, network depth provides much greater representational power relative to width. Intuitively, this makes sense – depth allows us to build hierarchies and stack multiple non-linearities. Indeed, this is where we begin to see the outlines of what the answer looks like in practice – depth provides a clear, easy way to bake structure into our network.

So, the upshot from the theoretical perspective is that the universal approximation theorem answers in the affirmative; yes, you could reduce a complex network architecture to a single hidden layer network, but the basis for doing this is a bit naïve – it doesn't consider the efficiency of the network relative to the number of parameters.The two papers linked above imply this.

Practice

In practice, there are issues with performing such a reduction. To date, the most successful models have deep architectures. More importantly though, many of these models are specialized – we have CNNs for computer vision, RNNs for sequence-based data, etc. These architectures were conceived as a way to integrate knowledge about the structure of the problem into the structure of the network to improve learning of the desired representation, and this too requires depth.

In fact, last year a paper took this idea to the extreme. Weight agnostic neural networks are specifically constructed such that their architecture is in essence the representation with a single parameter (weight) that can be varied to generate multiple distinct models with different performance characteristics. The WANNs that were developed in this research weren't especially deep, but they also weren't single layered.

The upshot here is that, for complex problems, depth is necessary for efficient learning and gradient computation and that it can be used to encode problem structure into a network. While it may be possible to reduce a learned representation to a single hidden layer network, it's not inconceivable that you may end up with more parameters condensed into a single hidden layer with very high dimensional gradients. In terms of implementation, this isn't desirable.

| improve this answer | |
$\endgroup$
  • $\begingroup$ As far as I know using a complex model to train a simpler one has been done. Probably by Schmidhuber. I'm not sure they still do it. $\endgroup$ – BlindKungFuMaster Oct 31 '17 at 10:39
  • 1
    $\begingroup$ Also: One model to learn them all: arxiv.org/abs/1706.05137 $\endgroup$ – BlindKungFuMaster Oct 31 '17 at 10:45
  • $\begingroup$ Nice, I'll have to read that paper – looks interesting. $\endgroup$ – Greenstick Oct 31 '17 at 16:44
  • $\begingroup$ How does this answer the main question? $\endgroup$ – nbro Feb 23 at 21:07
  • $\begingroup$ @nbro Go look at the original question; it asked whether you could reduce, for example, an RNN to an MLP. This answer points out how the architecture often encodes much of the function, providing evidence towards the negative. An extreme example of this would be weight agnostic neural networks (WNNs) that the Google AI team published last year. Put another way, while the universal approximation theorem supports the idea that you can reduce any architecture that encodes a continuous function to a single later neural net, this likely isn’t the case in practice. $\endgroup$ – Greenstick Feb 24 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.