MobileNet validation loss not decreasing over time

Question

I am trying to train a MobileNetV2 on a custom dataset, to image Classification task. Cardinality is 864 images, split in 70%/20%/10%, balanced between the 3 different classes.

Weights are pre-loaded from imagenet, I froze the net and I added to the bottom of the net a GlobalAveragePooling, a Dropout (with 50% drop probability), and a Dense layer with 3 classes and softmax as activation function, since i want the output layer to give me an output like (1,0,0) if the inference image is from the first class, and so on.

• image size: 96x96 (I normalized, too)
• batch_size: 32
• Learning rate: 0.001
• trainable params: 3843
• optimizer: sgd ('adam' doesn't improve my accuracy)
• loss: categorical cross entropy
• metrics: accuracy

Training for 20 epochs gives me these results:

After that I decided to try some fine-tuning, by freezing only the first 100 layers of the net. Trained again for 10 epochs, that's what I get:

My net is overfitting, but I don't know why it's happening and what am I expected to do in order to improve my accuracy.

Edit: I also tried increasing dataset images with some source images or even with some data augmentation, up to more than 3K images, but it didn't work out at any rate.

2nd edit: I'm currently using a sgd optimizer with a decay rate of 1e-6. I've just added the stoch dept per each conv2D layer with linear decay surviv rate, up to 50%. I reached a 80% val_accuracy before early stopping. I suppose pretraining can't be adopted to my personal use case, so I'll be exploring some other architectures, hoping for better results.

Answer 1

Since you already tried to gather more data along with data augmentation, and dropout. I can suggest you to use:

• Weight decay, but in the form of AdamW (or similar for SGD) in which the "W" stands for decoupled weight decay update. In this way the magnitude of the penalty is both easier to control and interpret. I think you should use a coefficient of 1e-4 or larger.
• Label smoothing, with an $\epsilon=0.05$ or $0.01$.
• Stochastic depth: is a regularization technique specific for networks with skip connections, basically dropping a residual path with probability $p$ (think of it as a dropout over entire skip connections.)
• Advanced image augmentations: like training on random crops of your images (to increase their number; also be sure to not crop much otherwise the content will be changed), cutout (zero random portion of image), mixup (blend to random images and their classes as well), and cutmix (as before with also cutout.)

If the issue still persists, try to pre-train on a different dataset and/or use a different network architecture.