0
$\begingroup$

I've been working on learning about NLP via a beginners competition on Kaggle.

I first trained a model with an embedding layer and then a simple linear layer. I actually got way better than a flip of the coin with this model, so I decided to try to step it up with an LSTM.

What happened was that training loss decreased and then palteaued while validation loss never decreased at all.

In the case of overfitting, I would expect validation loss to decrease for a while but then either remain steady or perhaps even increase as the model starts to overfit.

I can't find any reason for the strange loss curves I'm seeing:

enter image description here enter image description here

What could cause such a phenomenon?

I would be happy to share my network architecture and training code if there isn't a straightforward answer (I know there usually isn't).

$\endgroup$
1
$\begingroup$

As you know, it would be hard to tell exactly what is going on without knowing more about the dataset.

However, a couple things come to mind:

  1. Did u correctly normalize by fitting the scaler only on the train dataset and then apply the same transform (using the the mean and variance from train set) to the test set.

  2. Is your dataset imbalanced? I have found the Python DataPrep tell useful for exploratory data analysis

$\endgroup$
1
  • $\begingroup$ I didn't normalize the data as it was a tokenized vocabulary (the Kaggle Disaster Tweets dataset). As far as an imbalance goes, it is but not by much. I used a stratified split to make sure the ratio of positive samples in both datasets were the same. $\endgroup$ Aug 17 at 11:44
0
$\begingroup$

The conclusion that I finally arrived at was that I don't have enough data. The reason I believe this is true is that I slowly lowered model complexity and watched as overfitting decreased and underfitting increased.

If the model was not overfitting, i.e. if the validation loss decreased along with the training loss, then the model underperformed. It was little better than a flip of a coin on binary classification.

Increasing model complexity at all from this point lead to immediate overfitting.

I believe that the solution is to find more data, or use a model that generalizes better with less data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.