How do neural networks learn specific features throughout the layers?

Question

I was reading about convolutional neural networks and I came across such an explanation:

Consider detecting features in human face. The earlier layers of neural networks learn coarse features such as edges in the images and the latent layers learn more complex ( finer) features such as eyes, nose and etc

I am wondering why this is a true statement, namely how can we know that a neural network first starts by learning primitive features and then learns complex features. Could you please explain?

Answer 1

The reason is that a layer can only learn a function based on it's inputs - that's clear. When you look at a convolutional neural network, each output pixel of an early layer only gets a very small fraction of the whole image as input. For example: If your first layer uses a 3 x 3 kernel, then each output only depends on only 9 pixels. Try detecting anything other then edges with that much information - it's not not possible. In other words: Looking at a 3 by 3 image you will be able to see an edge and its orientation, but not something like a nose.

In contrast, the later layers depend on a much larger portion of the image and can thus learn high-level features like a nose.

It's more a question of whats possible for the layers to learn and not a question of the network self-organizing it's resources. And besides just reasoning like that you can visualize what the kernels in the individual layers respond to. I found this question that also has an image with some further information on that.

EDIT In response to the first comment:

To further clarify how a CNN works, here is an example, where $X_0$ is the input image and $X_1$ is the output of the first CNN-Layer:

1. Each convolution in the first layer receives a small (e.g. 3 x 3) area of pixels of $X_0$ and provides an output pixel. Here it is only possible to detect simple features (edges / gradients) because of the reason stated above.
2. The output of the first Layer $X_1$ is again an image, with the following properties:
   • A single pixel represents a 3 x 3 area in the original input
   • Effectively $X_1$ is a map that encodes the location of the simple features that got detected.
3. Because of the above properties, applying another 3 x 3 convolution to $X_1$ causes that
   • That convolution effectively depends on a 5 x 5 area of $X_0$, i.e. a 3 x 3 area of $X_1$
   • That convolution responds to certain arrangements of these 1st-level features, e.g. two adjacent edges with different orientations are a corner. You can think of the CNN-layers as a hierarchy where initial layers provide basic features the next layer detects compositions of these, the next layer detects compositions of the compositions and so on.

is it like the many basic functions (like the edge detections ) are added together and then create a function that fires high values when a "nose" detected ?

So yes, you can say it like this. This is exactly where the thought of hierarchy fits in.