I am fine-tuning a VGG16 model on 20 classes with 500k images I was wondering how do you chose the size of the dense layer (the one before the prediction layer which has a size 20). I would prefer not to do a grid search seeing how long it take to train my model.

Also how many Dense layer should I put after my global average pooling ?

base_model = keras.applications.VGG16(weights='imagenet', include_top=False)

  x = base_model.output
  x = GlobalAveragePooling2D()(x)
  x = Dense(???, activation='relu')(x)
  x = Dropout(0.5, name='drop_fc1')(x)
  prediction_layer = Dense(class_number, activation='softmax')(x)

I haven't see particular rules about how its done, are there any ? Is it link with the size of the convolution layer ?


It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.

  • 1
    $\begingroup$ Is it based on your own experience something else ? and why would 2 layers of 512 be better than for example a layer of 4096 ? $\endgroup$ – Hadrien Berthier Mar 20 '19 at 11:22
  • $\begingroup$ Just common practice. Historically 2 dense layers put on top of VGG/Inception. It works, so everyone use it. Intuition behind 2 layers instead of 1 bigger is that it provide more nonlinearity. But it's not proven. Also the number of weights would be quite different. It's different balance between weights and input/output pipe size $\endgroup$ – mirror2image Mar 20 '19 at 13:22
  • $\begingroup$ okay thanks :) I was wondering because I saw that a lot but didn't see anything written about $\endgroup$ – Hadrien Berthier Mar 20 '19 at 14:03

I am also wondering about this. It must depend both on convolutional sub-network output size (N) and number of classes (M). Maybe there are some rules of thumbs depending on (N, M).

  1. Why 2 dense layers and not, say, 3 or 4 ?
  2. Is it better to have all dense layers (except last) the same size ? or decreasing ? or increasing ? or pyramidal ?
  3. Is it better to have small dense layers or larger ones with dropout between layers ?

And bonus question:

  1. Should we use batch normalization between dense layers ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.