2
$\begingroup$

I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?

$\endgroup$
2
  • $\begingroup$ I would highly recommend watching this introductory lecture by DeepMind x UCL (34:00 - 40:00) which explains the Universal Approximation Theorem in an intuitive way. $\endgroup$
    – orion_ix
    May 20 at 22:36
  • $\begingroup$ @orion_ix wow, that video is fantastic! It explained why we go deep instead of wide in a very insightful way thanks for recommendation! $\endgroup$ May 21 at 0:31
2
$\begingroup$

This is akin to asking "Why do we need more than one instance of sine to represent any repeating function" or "why can't we represent any polynomial with an equivalent polynomial of just the first degree?" There are many, many problems... I'd even want to say most... that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer. This is not to say that the theorem is wrong, but consider the applied aspects. We can approximate any continuous function, but that might require a single layer that is infinitely wide, however that same function might be approximated by a deep network having only a few dozen neurons.

However, this is not to say that many networks could not be represented with networks that perform at least as well, or perhaps even better, by simpler networks of fewer layers/neurons. There is active research into how to generalize this.

Ultimately, non-trivial problems often require an empirical approach in this space currently because there is no general solution to "learning."

$\endgroup$
8
  • 2
    $\begingroup$ I think what you have answered is not correct in fact due to universal approximation theorem, we can approximate any higher dimensional function with only one hidden layer. $\endgroup$ May 21 at 10:36
  • 2
    $\begingroup$ @prakharnigam I think that the idea that this answer tries to convey is correct. You may be able to approximate your function with only one sine-like function by increasing e.g. the frequency, but you may do that more efficiently with a combination of multiple functions (and that's why using always a 1-hidden layer net is not typically done). Note that the universal approximation theorem (by Cybenko) tells you that you can approximate any continuous function, but provided that the number of neurons in the hidden layer is sufficiently large. $\endgroup$
    – nbro
    May 21 at 11:21
  • $\begingroup$ Yes, that's my point @nbro , thanks. $\endgroup$ May 21 at 11:23
  • $\begingroup$ @nbro I was asking, whether increasing layers is more efficient or adding neurons and why. That answer was a very generic one, of course i know what he is trying to convey but it is not rigorous and incorrect in some parts like saying " that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer", well every higher dimensional relationship that is continuous can be represented by one hidden layer. $\endgroup$ May 21 at 11:34
  • 1
    $\begingroup$ @prakharnigam I just read a paper last week that I'm having trouble finding that might be helpful as well... The topic was a new approach toward leaf and edge pruning in networks for graph connection pruning of weights and neuron pruning.. This is connected to the tendency for only a handful of neurons and the connections to following layers driving the majority of the accuracy in a network.. That isn't your question, but I suspect you'll find it is very much related. $\endgroup$ May 21 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.