# How to chose dense layer size?

I am fine-tuning a VGG16 model on 20 classes with 500k images I was wondering how do you chose the size of the dense layer (the one before the prediction layer which has a size 20). I would prefer not to do a grid search seeing how long it take to train my model.

Also how many Dense layer should I put after my global average pooling ?

base_model = keras.applications.VGG16(weights='imagenet', include_top=False)

x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(???, activation='relu')(x)
x = Dropout(0.5, name='drop_fc1')(x)
prediction_layer = Dense(class_number, activation='softmax')(x)


I haven't see particular rules about how its done, are there any ? Is it link with the size of the convolution layer ?

It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.

• Is it based on your own experience something else ? and why would 2 layers of 512 be better than for example a layer of 4096 ? – Hadrien Berthier Mar 20 '19 at 11:22
• Just common practice. Historically 2 dense layers put on top of VGG/Inception. It works, so everyone use it. Intuition behind 2 layers instead of 1 bigger is that it provide more nonlinearity. But it's not proven. Also the number of weights would be quite different. It's different balance between weights and input/output pipe size – mirror2image Mar 20 '19 at 13:22
• okay thanks :) I was wondering because I saw that a lot but didn't see anything written about – Hadrien Berthier Mar 20 '19 at 14:03

I am also wondering about this. It must depend both on convolutional sub-network output size (N) and number of classes (M). Maybe there are some rules of thumbs depending on (N, M).

1. Why 2 dense layers and not, say, 3 or 4 ?
2. Is it better to have all dense layers (except last) the same size ? or decreasing ? or increasing ? or pyramidal ?
3. Is it better to have small dense layers or larger ones with dropout between layers ?

And bonus question:

1. Should we use batch normalization between dense layers ?