# What is the concept of Tensorflow Bottlenecks?

What is the concept and how does one calculate Bottleneck values? How do these values help image classification? Please explain in simple words.

The bottleneck in a neural network is just a layer with less neurons then the layer below or above it. Having such a layer encourages the network to compress feature representations to best fit in the available space, in order to get the best loss during training.

In a CNN (such as Google's Inception network), bottleneck layers are added to reduce the number of feature maps (aka "channels") in the network, which otherwise tend to increase in each layer. This is achieved by using 1x1 convolutions with less output channels than input channels.

You don't usually calculate weights for bottleneck layers directly, the training process handles that, as for all other weights. Selecting a good size for a bottleneck layer is something you have to guess, and then experiment, in order to find network architectures that work well. The goal here is usually finding a network that generalises well to new images, and bottleneck layers help by reducing the number of parameters in the network whilst still allowing it to be deep and represent many feature maps.

• Hello Neil! I have a very simple question I think, but I am still confused. ai.stackexchange.com/questions/21172/… Do you have time to provide me an answer? – jgauth May 14 at 22:06
• @jgauth: I subscribe to the RSS feed of all questions on this site, and I attempt to answer all questions that I see where I think that I can. There is no need to contact me directly to have a question looked at. If you do want to contact me about something specific, it is best done on a related question or in the site chat if there isn't one. – Neil Slater May 15 at 7:35
• Hi @NeilSlater, how is a bottleneck layer different from a bottleneck feature? I have read some kernels on Kaggle and here where people use a pretrained Resnet (after removing its last few layers). The model is made to generate the features without any further training. The features are then passed to a classifier of their choice, usually an SVM or a simpler CNN to learn from those features. Clearly in that case, the network will not learn to compress the feature representations in the available space. – Anshuman Kumar Jun 2 at 14:56
• Like here the person uses ResNet kaggle.com/paperboiii/one-class-classification-for-images or here towardsdatascience.com/… – Anshuman Kumar Jun 2 at 14:59
• @NeilSlater, How much of a difference does compressing feature representations make to the overall accuracy? – Anshuman Kumar Jun 2 at 15:00

Imagine, you want to re-compute the last layer of a pre-trained model :

Input->[Freezed-Layers]->[Last-Layer-To-Re-Compute]->Output


To train [Last-Layer-To-Re-Compute], you need to evaluate outputs of [Freezed-Layers] multiple times for a given input data. In order to save time, you can compute these ouputs only once.

Input#1->[Freezed-Layers]->Bottleneck-Features-Of-Input#1


Then, you store all Bottleneck-Features-Of-Input#i and directly use them to train [Last-Layer-To-Re-Compute].

Explanations from the "cache_bottlenecks" function of the "image_retraining" example :

Because we're likely to read the same image multiple times (if there are no distortions applied during training) it can speed things up a lot if we calculate the bottleneck layer values once for each image during preprocessing, and then just read those cached values repeatedly during training.

Tensorflow bottleneck is the last pre prosessing phase before the actual training with data recognitions start. It is a phase where a data structure is formed from each training image that the final phase of training can take place and distinguish the image from every other image used in training material. Somewhat like a fingerprint of the image.

It is involved to the re-training command and as the name suggests, this is the main time consumer of the command execution. The amount of training material may have to be compromized if this bottleneck seems too time consuming.

As it is a command line command, I don't know the exact algorithm. Algorithm is public in code in Github but is supposedly so complicated (execution time is very long by definition) that I believe I cannot just write it down in this type of answer.