I was training a CNN model on TensorFlow. After a while I came back and saw this loss curve:

training loss curve(green) and validation loss curve(gray)

The green curve is training loss and the gray one is validation loss. I know that before epoch 394 the model in heavily overfitted, but I have no idea what happened after that.

Also, this is accuracy curves if it helps:

accuracy curves

I'm using categorical cross-entropy and this is the model I am using: model architecture

and here is link to PhysioNet's challenge which I am working on: https://physionet.org/content/challenge-2017/1.0.0/

  • $\begingroup$ Maybe you should provide a little bit of context, i.e. which task you are trying to solve with CNNs (i.e. which dataset), which loss function you are using, maybe you should do plot_model(your_model) and report the architecture of your model. $\endgroup$ – nbro Dec 14 '20 at 10:22
  • $\begingroup$ I'm trying to classify ECG signals in PhysioNet 2017 challenge. My output is four classes and I'm using categorical cross-entropy as loss function. $\endgroup$ – Sepehr Golestanian Dec 14 '20 at 10:35
  • $\begingroup$ Please, edit your post to include these details (maybe also a link to the challenge). $\endgroup$ – nbro Dec 14 '20 at 10:39
  • 1
    $\begingroup$ I've edited the question. $\endgroup$ – Sepehr Golestanian Dec 14 '20 at 10:59
  • $\begingroup$ Why are you using a CNN to classify ECG signals? Aren't ECG signals just numerical time series? $\endgroup$ – nbro Dec 14 '20 at 11:08

Follow this answer from StackOverflow, I think your problem related to the second cases where your loss gets Nan value.

Maybe you should try to use the larger datatype (for example float 16 -> float 32)

  • $\begingroup$ As I'm using tensorflow's built-in categorical cross-entropy, I think tf's team knows they should take good care of NaN, values. By the way, can regularization help in this situation or should I put weight constraints? $\endgroup$ – Sepehr Golestanian Dec 14 '20 at 10:45
  • $\begingroup$ We have to dig deeper into TF code to get the final conclusion. For the second question, I think yes, regularization is to solve the overfitting problem, here, before epoch 394, your validation accuracy is stable so I can say your model get overfit problem $\endgroup$ – CuCaRot Dec 14 '20 at 11:28
  • $\begingroup$ Thanks very much, I'll try this and update the question! $\endgroup$ – Sepehr Golestanian Dec 14 '20 at 12:51
  • $\begingroup$ @SepehrGolestanian So, why did you accept this answer? What was the problem in the end? $\endgroup$ – nbro Dec 15 '20 at 17:51
  • $\begingroup$ @nbro, you look like the admin of this community, I saw you carefully edit nearly all the post and ask in very details :smile: $\endgroup$ – CuCaRot Dec 16 '20 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.