Let's consider a deep convolutional network. It seems that there is some consensus on the following notions:

1. Shallow layers tend to recognise more low-level features such as edges and curves.

2. Deeper layers tend to recognise more high-level features (whatever this means).

While I usually come across various online articles and blogs that state this, no one ever cites literature that supports this claim. I am not seeking the question as to why this phenomenon happens, I'm only seeking whether it has actually been experimented on and documented. Also, I am barely able to find any peer-reviewed literature that provides evidence of this on sites such as Google Scholar or ResearchGate.

Could anyone point me to the right direction?

  • 1
    $\begingroup$ arxiv.org/abs/2106.14587 is the most exhausting, most fundamental work on this - it connects the architecture of NN with neural manifolds (manifolds of activities) and with categories of theories and languages. All is one system, layers (graphs, neural manifolds, languages-theories) are category theoretic fibrations. Just define learnability meatrics on this structurue and this can solve the problem of finding the optimal architecutre for the task. $\endgroup$
    – TomR
    Aug 22, 2022 at 7:45

2 Answers 2


It is assumed that NNs build up a hierarchical representation, whereby each layer combines features from the lower-level layers. The layers could be understood as representing a cascade of stacked features:

edges -> texture -> patterns -> parts -> objects

So from lower-level patterns to the more abstract higher-level concept like representation. This Distill article as far as I can tell is one of the most cited sources (740 citations) and provides an in-depth explanation of the features and how to visualize them. The journal is peer-reviewed.

The post also points to some older references such as: this, this or this. The website of Chris Olah one of the authors of the Distill article is also a great source for finding visualizations for different deep learning architectures.


You won’t find literature on this point because it’s true by definition. Low-level features are simple statistics of the raw input. High-level features are statistics of lower-level features. In a convolutional (or any feedforward) network the shallowest layers compute statistics directly on the input, so they create the lowest-level features. Deeper layers operate on the features of shallower layers, so they create higher-level features.

As an example, edges might be computed at the lowest/first convolutional layer, then corners at the second layer by looking for two perpendicular edges. The highest level may detect, say, whole faces, which is a complex, aka high-order, statistic.

If you want a visual example of what ‘low-level’ and ‘high-level’ features look like in a convolutional network, check out Google’s Deep Dream Generator, which emphasizes what the different layers ‘see’.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .