Val loss doesn’t decrease after a certain number of epochs

Question

I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use nn.CrossEntropyLoss() as my loss function. This is my network’s configuration

Model(
  (fc): Sequential(
    (0): Linear(in_features=800, out_features=1024, bias=True)
    (1): ReLU()
    (2): Linear(in_features=1024, out_features=512, bias=True)
    (3): ReLU()
    (4): Linear(in_features=512, out_features=256, bias=True)
    (5): ReLU()
  )
  (lstm): LSTM(256, 1024, bidirectional=True)
  (hidden2tag): Sequential(
    (0): Linear(in_features=2048, out_features=1024, bias=True)
    (1): ReLU()
    (2): Linear(in_features=1024, out_features=500, bias=True)
  )
)

This is what my loss looks like. It increases after reaching a particular value no matter what setting I use. I’ve used k-fold cross–validation but the loss/accuracy across all folds stays the same, so I’m assuming that there’s no issue in the distribution of train/val splits.

From left(Iteration-wise loss (train loss recorded after every batch), train loss (recorded after every train epoch), validation loss (recorded after every val epoch))

I’ve tried using step-wise LR scheduling (tried OneCycleLR and Multiplicative LR as well) but the loss still doesn’t improve

I’ve tried using dropout but that gives higher validation loss values instead

What can be done here in order to further decrease the loss?

Answer 1

The training loss is continually decreasing, but the the validation loss shows a local minimum at epoch ~100 which indicates that there is overfitting starting to happen. This means that your model has sufficient capacity to learn (i.e., it is not underfitting) and that increasing the model's capacity (i.e., increasing the number of layers or increasing the number of neurons per layer) is unlikely to help.

Instead, you should work towards reducing overfitting. The best way to do this is by collecting more real data. You can also consider Data Augmentation or simulating synthetic data. Other approaches such as dropout and L1 and/or L2 regularization may also be helpful. This article "An Overview of Regularization Techniques in Deep Learning (with Python code)" is a nice overview of these methods. Even though you have used dropout, you can tune the dropout rate and include dropout at one or more layers in your model.

Finally, ensemble methods have shown success in improving accuracy in many (but not all) settings, compared to single models.

Answer 2

I fully agree with @Snehal Patel's comment but want to expand on the capacity of the model a bit.

The easiest approach to reduce overfitting is to reduce the capacity of the model, i.e. the size of its layers.

Therefore, in your first sequential model, instead of three linear with [1024, 512, 256] nodes, it would probably be better to have just two with [512, 256] nodes.

The same applies to the second sequential model. Just using one linear layer probably reduces the capacity of the model to overfit to the data.

I hope those changes work out for you.