Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Avoid representational bottlenecks, especially early in the
network. Feed-forward networks can be represented by an acyclic graph
from the input layer(s) to the classifier or regressor. This defines a
clear direction for the information flow. For any cut separating the
inputs from the outputs, one can access the amount of information
passing though the cut. One should avoid bottlenecks with extreme
compression. In general the representation size should gently decrease
from the inputs to the outputs before reaching the final
representation used for the task at hand. Theoretically, information
content cannot be assessed merely by the dimensionality of the
representation as it discards important factors like correlation
structure; the dimensionality merely provides a rough estimate of
This paragraph warned us to avoid bottlenecks and also representational bottlenecks. What does it mean by a bottleneck of/in a neural network and representational bottleneck?
A bottleneck layer is a layer that contains few nodes compared to the previous layers. It can be used to obtain a representation of the
input with reduced dimensionality.
If you create bottlenecks in the initial layers it can cause loss of information.
Why it is done?
Take the example of an image dataset that contains high-resolution images. High resolution means more pixels means it will need more nodes in the input layer.
Having more nodes will need more computational power to train the network. Hence in such cases, we can use fewer nodes in the next layer. As images are high resolution we might not lose any important information.