2
$\begingroup$

I trained a neural network on the UNSW-NB15 dataset, but, during training, I am getting spikes in the loss function. The algorithms see part of this UNSW dataset a single time. The loss function is plotted after every batch.

enter image description here

For other datasets, I don't experience this problem. I've tried different optimizers and loss functions, but this problem remains with this dataset.

I'm using the fit_generator() function from Keras. Is there anyone experience this problem using Keras with this function?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The spikes could be caused by many reasons: insufficient model capacity, incorrect label, buggy input parsing, ... Finding out the culprit requires some detective work. For instance, you could apply the learned model to the whole train set and manually examine the datapoints which result in the highest loss. Alternatively, you could compare the learning outcomes of different models (both weaker and stronger).

Improve this answer
$\endgroup$
0
$\begingroup$

Andrew Ng explains with great details in Deep Learning Course appears in the image below. He also focuses on some corners can cause this problem :

  1. some mislabeled examples in the dataset.
  2. the size of your mini batch or change it with GD.

enter image description here

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ This is not the same as this case, the mean of spikes in Andrew Ng's picture is pretty much a smooth curve, whereas the mean in OP's graph it is irregular. $\endgroup$
    – user9947
    Mar 23, 2019 at 19:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .