I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use
nn.CrossEntropyLoss() as my loss function. This is my network’s configuration
Model( (fc): Sequential( (0): Linear(in_features=800, out_features=1024, bias=True) (1): ReLU() (2): Linear(in_features=1024, out_features=512, bias=True) (3): ReLU() (4): Linear(in_features=512, out_features=256, bias=True) (5): ReLU() ) (lstm): LSTM(256, 1024, bidirectional=True) (hidden2tag): Sequential( (0): Linear(in_features=2048, out_features=1024, bias=True) (1): ReLU() (2): Linear(in_features=1024, out_features=500, bias=True) ) )
This is what my loss looks like. It increases after reaching a particular value no matter what setting I use. I’ve used k-fold cross–validation but the loss/accuracy across all folds stays the same, so I’m assuming that there’s no issue in the distribution of train/val splits.
From left(Iteration-wise loss (train loss recorded after every batch), train loss (recorded after every train epoch), validation loss (recorded after every val epoch))
I’ve tried using step-wise LR scheduling (tried OneCycleLR and Multiplicative LR as well) but the loss still doesn’t improve
I’ve tried using dropout but that gives higher validation loss values instead
What can be done here in order to further decrease the loss?