2
$\begingroup$

I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?

Improve this question
$\endgroup$
2
  • $\begingroup$ I would highly recommend watching this introductory lecture by DeepMind x UCL (34:00 - 40:00) which explains the Universal Approximation Theorem in an intuitive way. $\endgroup$
    – orion_ix
    May 20, 2021 at 22:36
  • $\begingroup$ @orion_ix wow, that video is fantastic! It explained why we go deep instead of wide in a very insightful way thanks for recommendation! $\endgroup$
    – Prakhar
    May 21, 2021 at 0:31

1 Answer 1

Reset to default
2
$\begingroup$

This is akin to asking "Why do we need more than one instance of sine to represent any repeating function" or "why can't we represent any polynomial with an equivalent polynomial of just the first degree?" There are many, many problems... I'd even want to say most... that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer. This is not to say that the theorem is wrong, but consider the applied aspects. We can approximate any continuous function, but that might require a single layer that is infinitely wide, however that same function might be approximated by a deep network having only a few dozen neurons.

However, this is not to say that many networks could not be represented with networks that perform at least as well, or perhaps even better, by simpler networks of fewer layers/neurons. There is active research into how to generalize this.

Ultimately, non-trivial problems often require an empirical approach in this space currently because there is no general solution to "learning."

Improve this answer
$\endgroup$
8
  • 2
    $\begingroup$ I think what you have answered is not correct in fact due to universal approximation theorem, we can approximate any higher dimensional function with only one hidden layer. $\endgroup$
    – Prakhar
    May 21, 2021 at 10:36
  • 2
    $\begingroup$ @prakharnigam I think that the idea that this answer tries to convey is correct. You may be able to approximate your function with only one sine-like function by increasing e.g. the frequency, but you may do that more efficiently with a combination of multiple functions (and that's why using always a 1-hidden layer net is not typically done). Note that the universal approximation theorem (by Cybenko) tells you that you can approximate any continuous function, but provided that the number of neurons in the hidden layer is sufficiently large. $\endgroup$
    – nbro
    May 21, 2021 at 11:21
  • $\begingroup$ Yes, that's my point @nbro , thanks. $\endgroup$
    – David Hoelzer
    May 21, 2021 at 11:23
  • 1
    $\begingroup$ As I stated above, the UAT tells you that, in theory, you can approximate a continuous function with a 1-hidden layer (with an arbitrary width) neural network, but it doesn't tell you how to do it in practice. I don't know the exact mathematical answer to your question, but we can say that, in practice, in many cases, neural networks with more than 1 hidden layer are used (probably because they are more effective). You may be interested in this post (and the related/linked posts and papers). $\endgroup$
    – nbro
    May 21, 2021 at 11:56
  • 1
    $\begingroup$ @prakharnigam I just read a paper last week that I'm having trouble finding that might be helpful as well... The topic was a new approach toward leaf and edge pruning in networks for graph connection pruning of weights and neuron pruning.. This is connected to the tendency for only a handful of neurons and the connections to following layers driving the majority of the accuracy in a network.. That isn't your question, but I suspect you'll find it is very much related. $\endgroup$
    – David Hoelzer
    May 21, 2021 at 14:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .