What is the purpose of the hidden layers?

Question

Why would anybody want to use "hidden layers"? How do they enhance the learning ability of the network in comparison to the network which doesn't have them (linear models)?

Answer 1

"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the question isn't really "How do hidden layers help?" as much as "Why are deeper networks better?".

And the answer to that latter question is an area of active research. Even top experts like Geoffrey Hinton and Andrew Ng will freely admit that we don't really understand why deep neural networks work. That is, we don't understand them in complete detail anyway.

That said, the theory, as I understand it goes something like this... successive layers of the network learn successively more sophisticated features, which build on the features from preceding layers. So, for example, an NN used for facial recognition might work like this: the first layer detects edges and nothing else. The next layer up recognizes geometric shapes (boxes, circles, etc.). The next layer up recognizes primitive features of a face, like eyes, noses, jaw, etc. The next layer up then recognizes composites based on combinations of "eye" features, "nose" features, and so on.

So, in theory, deeper networks (more hidden layers) are better in that they develop a more granular/detailed representation of a "thing" being recognized.

Answer 2

Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of linear layers down to a single layer.

This is why we use nonlinear activation functions, like RELU. This allows us to add a level of nonlinear complexity with each hidden layer, and with arbitrarily many hidden layers we can construct arbitrarily complicated nonlinear functions.

Because we can (at least in theory) capture any degree of complexity, we think of neural networks as "universal learners," in that a large enough network could mimic any function.

Answer 3

Actually, the hierarchical learning explanation given by mindcrime is not that acceptable anymore (This was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, and this explanation does not make sense for such neural networks. However, we can think of it as solving the knots of high dimensional manifolds, i.e. we transform the input into high dimensional space, and this helps us to find a better representation of the data.

A geometric interpretation was explained as such in the book Deep Learning with Python by François Chollet:

...you can interpret a neural network as a very complex geometric transformation in a high-dimensional space, implemented via a long series of simple steps...

Imagine two sheets of colored paper: one red and one blue. Put one on top of the other. Now crumple them together into a small ball. That crumpled paper ball is your input data, and each sheet of paper is a class of data in a classification problem. What a neural network (or any other machine-learning model) is meant to do is figure out a transformation of the paper ball that would uncrumple it, so as to make the two classes cleanly separable again. With deep learning, this would be implemented as a series of simple transformations of the 3D space, such as those you could apply on the paper ball with your fingers, one movement at a time. Uncrumpling paper balls is what machine learning is about: finding neat representations for complex, highly folded data manifolds. At this point, you should have a pretty good intuition as to why deep learning excels at this: it takes the approach of incrementally decomposing a complicated geometric transformation into a long chain of elementary ones, which is pretty much the strategy a human would follow to uncrumple a paper ball. Each layer in a deep network applies a transformation that disentangles the data a little—and a deep stack of layers makes tractable an extremely complicated disentanglement process.

I suggest you to read this brilliant blog post to learn about the topological interpretation of deep learning.

Also, this toy interactive code may help you.

In the context of machine learning, the concept of a manifold can be illustrated as in the following figure.

In the first part, data are 3-dimensional. However, we can find a transformation to get the second image, which shows that data is actually artificially high dimensional, i.e. it is a 2-dimensional manifold in 3-D space. This example may be thought of as a classification problem, and colors may represent classes, and we can find a trivial representation of the data for classification.

Another example could be following figures from the blog I mentioned. In here, this classification problem cannot be solved without having a layer that has 3 or more hidden units, regardless of depth. So the notion of high dimensional transformation is important.

We can map this data to 3-D, and find a plane to separate them.

Answer 4

One aspect that I'd like to add to the previous answers is the so-called Curse of dimensionality. This concept refers to the problem that many algorithms have a time complexity that grows exponentially with the dimension of the data.

As a simple example, let us consider a set $\{0,1\}^{D}$ that has only two values per dimension. For example, $\{0,1\}^{2} = \{(0,0),(0,1),(1,0),(1,1)\}$ and $(0,1,0) \in \{0,1\}^{3}$. Now imagine that you are given a function $f: \{0,1\}^{\times D} \rightarrow \{TRUE, FALSE\}$ that outputs TRUE exactly for one particular input. The goal is to determine that input.

In the example, if nothing else is known about f, the best thing one can do is to try the inputs one after another. However, $\{0,1\}^{D}$ has $2^D$ elements. So the number of inputs one has to try out will in general be roughly $2^D$ as well.

However, there exist examples suffering from a curse of dimensionality that can be solved with deep learning, i.e. using neural networks with many hidden layers. One example of great practical importance is given by high-dimensional partial differential equations, see e.g this report:

http://www.sam.math.ethz.ch/sam_reports/reports_final/reports2017/2017-44_fp.pdf

or this example for heat equations:

https://arxiv.org/abs/1901.10854

I also found this review on using deep learning to overcome the curse of dimensionality:

https://cbmm.mit.edu/sites/default/files/publications/02_761-774_00966_Bpast.No_.66-6_28.12.18_K1.pdf